Jump to content

Experimenting with ReFS on External USB Backup Drive


NoelC

Recommended Posts

That (devices been much smarter than what they appear from the outside) is a known (you decide) issue or feature, that has been noticed since the advent of IDE disks and exacerbated by the the "solid state devices", beginning with flash  disks and CF cards, but easily extendable to SSD's.

The "feature" side is that everything is managed by the hardware+firmware, and as such it is undoubtedly faster and more reliable (as long as the firmware is well programmed/stable).

 

The "issue" side is that since a few years we are the proud owners of "black boxes" which have completely undocumented internal behaviours and that can go "beserk" (by pure chance) in which case your only chances to recover data is through the services of the manufacturer of the "black box" (when and if they provide such services) or through the use of specialized tools that are nothing but (well done and "high quality level") a form of reverse engineering or - in some cases - pure "hacks".

 

A hard disk (or USB bridge, SSD, etc. ) controller is to all effects a rather powerful computer in itself, see this interesting hack:

http://spritesmods.com/?art=hddhack

that exposes in an unprecedented way the capabilities of a common (not particularly new) hard disk.

 

Till now we had thus a "dumb" filesystem and a "smart" device.

 

With SSD's we have even "smarter" devices. (at least in theory)

 

If you apply to SSD's a "smart filesystem" there is the risk that the two intelligences will come into a conflict of some kind. :ph34r:

 

From time to time I have nightmares about these families of disks :w00t: malfunctioning (that hopefully have not been "common" and I believe are not even manufactured anymore):

http://www.forensicfocus.com/Forums/viewtopic/p=6560563/

or however (hopefully) coinfined to niches like security/intelligence, wtc.

http://www.pcworld.com/article/225202/Toshibas_New_Self_Erasing_Hard_Drives_The_Ultimate_in_Data_Security.html

http://storage.toshiba.com/storagesolutions/trends-technology/security-hard-drives

as well as those nightmares about BYOD and remote wiping of devices :ph34r::

http://www.msfn.org/board/topic/163387-byod-ideas-opinions-whatever/

http://www.forensicfocus.com/Forums/viewtopic/t=10567/

 

jaclaz

Link to comment
Share on other sites


Well, knock on silicon, I've been relying on 4 SSDs in RAID 0 since April 2012 and they haven't even hinted at a single glitch.  I am thinking I accidentally coupled the right level of in-box smarts with on-RAID-card smarts with file system smarts.  :)

 

-Noel

Link to comment
Share on other sites

Thanks for the information about storage spaces, they are available in Windows 8 Home (or just Windows 8 in contrast with Windows 8 Pro) but you need some more drives. I suppose that you can use virtual drives for the job too but I don't have much initiative to experiment on it at the moment.

Link to comment
Share on other sites

  • 2 weeks later...

Just to follow up, my ReFS formatted drives (one internal, one external) have been working flawlessly, going through terabytes of backup data without a glitch night after night.

 

-Noel

Link to comment
Share on other sites

  • 2 weeks later...

I've read that the system maintains the ReFS system, including checking for "bit rot" (degradation of data stored on the disk but which is not accessed for a long time).

 

Today I caught the System reading all the files on my backup drive.  It appears it was doing a (scheduled?) check for bit rot, just as documented.

  • The activity was not intrusive in any way.  Only reason I noticed it was because of some fairly relaxed seeking sounds from my backup drive.
     
  • My USB connection is capable of carrying about 60 MB/second, based on observations of deliberate copies of large files.  Yet this activity was only about 15 to 20 MB/second.
     
  • It wasn't sending the data anywhere, just reading it.

 

I think it's pretty cool to see the new technology in action.

 

-Noel

Link to comment
Share on other sites

  • 7 months later...

This is very informative thank you for the overview, there is surprisingly little good information on REFS on the web.

Will the self healing feature of REFS function when it is used on a removable device?

Is there any benefit to running REFS on a raid5 array, or would that be entirely redundant?

It seems that the storage spaces parity is in some ways superior to RAID in fault tolerance minus the performance.

 

Thanks 

Edited by techswine
Link to comment
Share on other sites

I should think it'd work fine on an external drive, but honestly I don't know for sure how much if any of the self-healing functionality has been exercised - my ReFS-formatted drives both work as well as the day I started using them, so if there has been any recovery activity, I just haven't noticed.  I do glance at my System Event Log from time to time, and I haven't seen anything listed there.

 

That might say there's a lack of management information and control with ReFS - or that I just haven't found it.  ReFS management panels might be found in server management packages I don't normally use.  Certainly Microsoft didn't / doesn't expect us to run ReFS in Win 8+, even though it runs perfectly as far as I can see.

 

How ReFS would work with RAID 5 et. al...  I haven't any anecdotal information, so I can't say.  Since the two systems work at different levels it seems as if they should complement each other nicely.  Whether there would be gotchas or thrashing situations I can't say.  I would certainly want to test it on an array before committing real data to it.

 

One other thing, re: Performance...

 

Both my internal HDD and my external USB drive that are ReFS-formatted have been taking incremental file backup updates for quite a long time now, and they're always nearly full, yet I've not noticed any performance degradation at all.  Performance is holding up remarkably well, actually.

 

I'm disappointed to see that there appears to be ZERO advancement of ReFS in Windows 10.

 

-Noel

Link to comment
Share on other sites

 

I'm disappointed to see that there appears to be ZERO advancement of ReFS in Windows 10.

 

AFAIK it is still in testing stages

also by analysis of other IT people, it seems ReFS is crippled compared to NTFS

its only advantage is that self-healing thing that nobody cares about :P

Link to comment
Share on other sites

As a digital photographer with terabytes of RAW and JPG2000 photos archived over the years, I began to realize that NTFS isn't as bitperfect as I used to believe.

A common phenomenon with JPG2000 and some lossless image compression file formats was distorted stripes in stored images over the years. I wouldn't believe it unless I experienced it myself.

It seems like some additional ecc layers are needed for long term storage and from my reading it appears the self healing function of REFS is approaching this.

I am on RAID5 after losing a lot of good work but it has it's own inherent vulnerabilities (write hole) which from my reading it appears storage spaces entirely circumvents.

Link to comment
Share on other sites

My understanding is that the ReFS implementation is supposed to embark on the recovery activities itself, absolving the user from having to mess with it.  Whether that's comforting or disturbing...  All I can say is on both internal and external drives I haven't encountered any problems yet.

 

I buy decent hardware that doesn't often fail, so mostly my interest was/is the performance.  It's not really noticeably better, except as I mentioned it seems not to get as balky when the drive is nearly full (who has a drive that's not normally nearly full?).

 

What I was hoping was that by the time of the next OS (e.g., Win 10) they'd make it possible to format the boot/system volume (C:) with ReFS.  At this point I've not fooled with ReFS on SSD hardware yet - I believe I'll do that soon.  It will be interesting since when the hardware gets more efficient the differences in the implementation of the file system become a bigger factor in the performance.

 

-Noel

Link to comment
Share on other sites

It will be interesting since when the hardware gets more efficient the differences in the implementation of the file system become a bigger factor in the performance.

... or the differences in the actual OS drivers for the filesystems, unrelated :w00t::ph34r:, but JFYI :):

http://www.msfn.org/board/topic/125116-fat16-vs-fat32-vs-ntfs-speed-on-usb-stick/

 

jaclaz

Link to comment
Share on other sites

I do know what you mean.  Drivers matter.

 

The RAID controller card I run (Highpoint 2720SGL) was very innovative but got an initial bad rep because of the drivers it shipped with.  Basically they had been written in the time of HDDs, not SSDs, so when used with SSDs the performance was essentially dismal.

 

Then they rewrote the drivers and lo and behold the performance skyrocketed.  But it was too late.  The Internet reviews were out.  It was not the thing to buy.

 

All that conspired to make the price initially low and was of course good for those of us who paid attention to the later figures (e.g., here).

 

What I don't know yet is whether ReFS will play well with this controller and its drivers.  I'm still short one SSD before implementing my planned expansion, in which I'll have all the drives in RAID 0 and set up an additional partition and data volume over and above the 2 TB volume I have now.  I believe I'll set that up, at least for initial testing, to be a ReFS formatted volume and see what happens.

 

-Noel

Edited by NoelC
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...