Data Management!

[JohnsonMiller]

Hi Guys,

For past three weeks, I am trying to recover from a big data lose. Apart technical reason, which still do not understand what was reason, I got a real feeling of data lose, what nightmare it is. And the other side, what a beautiful moment and calm mind, when you have a proper data back-up. I had a chance to taste both.

Long story short, I replaced my HDD with new SSD, then opening an old external storage, borrowed SATA to USB board and attached my HDD; everything was OK and started to MOVE data from HDD to SSD. After finishing the task, data was deleted from HDD but at SSD side almost every file was damaged. Focusing on Altium files, schematic file was there with exactly correct file size, but some connection or component was gone! The PCB file was exactly correct size, too, but Altium was unable to open it. Opening PRJ, there was tons of crash and error message from Altium each moment. In report MS WORD files, some page or figures was destroyed. Same for pages of PDF datasheets.
Very strange case was JPEG or PNG files, icon view of files was correct, but opening it the content was with color change or partly destroyed.
Still have question, what I did wrong?

Luckily, the HDD was not formatted, and it was possible to recover some of the deleted files, but the folder structure is gone. Now, I am examining project data file-by-file; a real painful task.

I am sharing this story, to worn how dangerous is trusting simple storage equipment. Also, inviting to share your experience, and help to possibly answer my question, what I did wrong?

Having this BAD experience, I also would like to start a discussion about how to keep data and when to issue version/revision? Do you recommend using SVN or other file versioning system?
Please share your opinion and experiences.

BR,

[robertferanec]

Ah, I hope you can recover everything.

The simplest solution which I use a lot (including my family photos and videos), I pay 9.99 for 2TB Google drive: https://one.google.com/home This way, local folder is automatically synchronized with online storage. Important - this will not safe you from deleting files (you may be able to recover some until 1 month, but not after that).

At company I use NAS server (I have simple TS-453A: https://www.amazon.com/QNAP-Professi.../dp/B017YB7T6U ). I have two disks mirrored (for important stuff and for clients who do not want to have projects in cloud) + 1 big HDD. Some important folders are automatically backed up to Google drive (this is in case of sudden lost), all the other folders and files are backed up once a week to Amazon glacier ( https://aws.amazon.com/glacier/ ) low cost archive storage. All these backups are supported by applications on the NAS server, you just need to set them up. All folders from NAS are mapped to local network - this makes it fast to work with the files and on background they are automatically backed up. Again, this will not help if you delete files by accident.

At company I also use Github (Team plan $25 / month = $300 / year for 5 people, now they have also Unlimited private repositories on the Free plan, maybe check it out: https://github.com/pricing ). Paying for this is one of the best things I have ever done. I use it a lot - mostly for software projects, but I do upload there now also my HW projects. The best is, even if you would delete something by accident on your hardrive, you can always go to github and find it there, including all the previous versions. It may be a little bit tricky to get used to work with git, but once you know how to use it, it is a great tool. Here is an example of our open source project: https://github.com/FEDEVEL/28pins

Important: I do not use Github with Altium as proper SVN integration. I tried SVN directly in Altium and it just didnt feel right for me. Instead, when I finish something important, I simply push the whole directory to github, that is how I use it. For everyday backup I relay on the Googledrive.

At this time, probably also other solutions exists, but I use google drive as it was one of the first things what existed when I needed it and I am happy with it - so no plans to change it for now.

I am also curious what other people use, how they backup their files and what is their experience with SVN. Please share. Thank you.

[JohnsonMiller]

Robert, Thank you for sharing your experiences and valuable comments,

To the best of my knowledge, SVN require a commit/check-out process; and each commit creates a sub-branch. After a small work or unreleased design, committing will make version number unnecessarily increasing. What do you recommend, when we should commit a design? and when release a design?

BR,

[robertferanec]

SVN require a commit/check-out process;

- that is why I do not use the SVN integrated feature inside of Altium .. just adds unnecessary steps and can make too many versions.

Simply, I installed git on my computer ( https://git-scm.com/downloads ) and when I am done with something important (e.g. finished with project, documents created, etc) only then I manually push it. I have not used SVN on my big projects yet (small projects I simply push when they are finished), but I would be probably committing the project before big changes and when the project would be moving to different stage (e.g. schematic finished ... schematic checked ... placement done ... all connected ... finished ...)

[Paul van Avesaath]

Robert, does altium work well wit github?? is it integrated,, or just as an external application.. ? looking for a good way to backup projects.. but I want to strip all the unnescesarry stuff automatically.. could be a good video (or maybe you did one already.. will check that)

[robertferanec]

I do not use Altium with github directly (I do not know if they work together). I manually sync my directories with Github, it looks like this:

git add .
git commit -m "What changes I have done"
git push origin master

The first line adds for commit all the new/updated files. The second line will commit the changes and add a description message. The third line will push all the changes from your PC to Github. Github also has a Windows application for this which you can try.