Jump to content

Welcome to MSFN Forum
Register now to gain access to all of our features. Once registered and logged in, you will be able to create topics, post replies to existing threads, give reputation to your fellow members, get your own private messenger, post status updates, manage your profile and so much more. This message will be removed once you have signed in.
Login to Account Create an Account



Photo

System Testing and Benchmarks


  • Please log in to reply
93 replies to this topic

#26
ptd163

ptd163

    Junior

  • Member
  • Pip
  • 51 posts
  • Joined 17-May 13
  • OS:Windows 7 x64
  • Country: Country Flag

Behold. The Windows Experience Index has returned.

 

http://winaero.com/c...omment.news.220




How to remove advertisement from MSFN

#27
DosProbie

DosProbie

    Friend of MSFN

  • MSFN Sponsor
  • 953 posts
  • Joined 16-October 12
  • OS:Windows 8.1 x64
  • Country: Country Flag

Donator

Behold. The Windows Experience Index has returned.

 

http://winaero.com/c...omment.news.220

Nice find! and fast also..DP :whistle:



#28
NoelC

NoelC

    Software Engineer

  • Member
  • PipPipPipPipPipPipPip
  • 1,805 posts
  • Joined 08-April 13
  • OS:Windows 8.1 x64
  • Country: Country Flag

Behold. The Windows Experience Index has returned.

 

http://winaero.com/c...omment.news.220

 

Neat.  I love it when people bring back things Microsoft deletes.  Only thing is I'm not sure I want a WEI score badly enough to trust it not to have a malware payload hidden somewhere inside.  I have stored copies of winsat output and can easily compare them looking for evidence of problems.

 

Microsoft probably figures the majority of people don't need to know about the assessment, since knowing causes some of them to try to game it for bragging rights, which could lead to maintenance problems.  Parts of the system do alter their behavior by the stored winsat results.

 

-Noel



#29
TELVM

TELVM

    Advanced Member

  • Member
  • PipPipPip
  • 349 posts
  • Joined 09-February 12
  • OS:Windows 7 x64
  • Country: Country Flag

"... the reason might be that Microsoft's own hardware was starting to look pretty bad using the Index. For example its expensive Surface Pro gets a 5.6 on the WEI scale when it was possible for a rubbish PC to get a score of 7.0 ..."

 

http://news.techeye....-its-dark-roots



#30
NoelC

NoelC

    Software Engineer

  • Member
  • PipPipPipPipPipPipPip
  • 1,805 posts
  • Joined 08-April 13
  • OS:Windows 8.1 x64
  • Country: Country Flag

I could believe that.

 

And let's not forget that the exact same hardware got higher scores with Windows 8.0's WEI measurement than with that of Windows 7.

 

-Noel



#31
DosProbie

DosProbie

    Friend of MSFN

  • MSFN Sponsor
  • 953 posts
  • Joined 16-October 12
  • OS:Windows 8.1 x64
  • Country: Country Flag

Donator

These are some Aero Glass benchmarks with 8.1 from my SSD drive using "AS SSD Benchmark" with Aero Enabled and also Aero Disabled.
I love Aero Glass but as you can see there is a significient difference anyone else have similiar results or comments?
 
~DP  :whistle:
 
ENABLED
2263_With_Aero.png
 
DISABLED
2888_No_Aero.png
 
 
 


#32
NoelC

NoelC

    Software Engineer

  • Member
  • PipPipPipPipPipPipPip
  • 1,805 posts
  • Joined 08-April 13
  • OS:Windows 8.1 x64
  • Country: Country Flag

Thing is, AS-SSD tends to vary a LOT from run to run anyway, which I've discussed above. That's one of the reasons I don't use it to try to detect system issues.

 
As mentioned earlier, also with Intel drivers your RAM write-back cache affects the numbers quite a lot in that case, and varying amounts of available RAM for cache at the instant you're doing the testing can cause even more variation in results.

 
Here are my AS-SSD results with Aero Glass on or off (no Intel RST drivers, nor RAM cache variations), as tested just now...

 
Aero Glass for Win 8.1 On:
ASSSD_With_AGW8.png

 

Aero Glass for Win 8.1 Off
ASSSD_Without_AGW8.png

 

As you can see, my results differ a fair bit from run to run as well - but in the opposite direction from yours.

 
-Noel



#33
DosProbie

DosProbie

    Friend of MSFN

  • MSFN Sponsor
  • 953 posts
  • Joined 16-October 12
  • OS:Windows 8.1 x64
  • Country: Country Flag

Donator

@Noel

Thanks for your post, maybe someone else can chime in and see what they get as well.

 

~DP :whistle:



#34
HarryTri

HarryTri

    Member

  • Member
  • PipPip
  • 211 posts
  • Joined 17-August 12
  • OS:Windows 8 x64
  • Country: Country Flag

 

Only thing is I'm not sure I want a WEI score badly enough to trust it not to have a malware payload hidden somewhere inside.  I have stored copies of winsat output and can easily compare them looking for evidence of problems.

 

I don't think that it is malware, as for the results and the rest stored information: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winsat


I always love Windows XP!


#35
DosProbie

DosProbie

    Friend of MSFN

  • MSFN Sponsor
  • 953 posts
  • Joined 16-October 12
  • OS:Windows 8.1 x64
  • Country: Country Flag

Donator

You can do a commandline but the gui is a nice touch as well..

With 8.1 "Programmers giveth as Microsoft taketh"

 

~DP :whistle:



#36
TELVM

TELVM

    Advanced Member

  • Member
  • PipPipPip
  • 349 posts
  • Joined 09-February 12
  • OS:Windows 7 x64
  • Country: Country Flag

Have a look at this beast  :ph34r:  :

 

DSC00944.JPG

 

Intel SSD DC P3700 800GB Review - Ludicrous Speed for the Masses!

 

Any chance NVMe might somehow work in W7?


  • DosProbie likes this

#37
NoelC

NoelC

    Software Engineer

  • Member
  • PipPipPipPipPipPipPip
  • 1,805 posts
  • Joined 08-April 13
  • OS:Windows 8.1 x64
  • Country: Country Flag

I don't see detail on how much of the low-level flash management is moved to the CPU.  Dedicated controllers do the job of managing the hardware inside the SSD boxes with today's models.  If NVMe moves much of that task to the driver, not only might that take resources away from your applications but it's not hard to imagine that data could be corrupted upon any PC instability.

 

Given that the market penetration percentage of Windows 8+ is deep in the single digits, TELVM, do you think Intel might be working on Windows 7 drivers for the thing?  :yes:  Bet on it.

 

I would like to see it compared to something like RAID 0 4 x 512 GB OCZ Vector or 4 x 512 GB Samsun 840 Pro SSD arrays using traditional SATA III.  I've seen benchmarks with such hardware that compete favorably with the numbers they're posting, and without quirks (such as 344 BYTES per second with 512 byte writes - what's up with that?).

 

The price seems pretty attractive, considering the $1.50 to $3,02 / GB range is list price for a newly announced part.  I wonder what they'll drop to after they ramp up production.

 

I'll bet server farm managers are salivating over this.

 

-Noel



#38
NoelC

NoelC

    Software Engineer

  • Member
  • PipPipPipPipPipPipPip
  • 1,805 posts
  • Joined 08-April 13
  • OS:Windows 8.1 x64
  • Country: Country Flag

I would like to see it compared to something like RAID 0 4 x 512 GB OCZ Vector or 4 x 512 GB Samsun 840 Pro SSD arrays using traditional SATA III. 

 

Over on anantech.com, they state:

 

"A single P3700 ends up replacing 4 - 6 high performance SATA drives."

 

-Noel



#39
TELVM

TELVM

    Advanced Member

  • Member
  • PipPipPip
  • 349 posts
  • Joined 09-February 12
  • OS:Windows 7 x64
  • Country: Country Flag

Ouch ouch. From Anandtech:

 

randomreadsm.png

 

"Small block random read operations have inherent limits when it comes to parallelism. In the case of all of the drives here, QD1 performance ends up around 20 - 40MB/s. The P3700 manages 36.5MB/s (~8900 IOPS)"

 

 

^ This breaks the party as far as I'm concerned. Any good SATA III SSD can manage those 4K reading speeds @ QD1 .

 

There seems to be some sort of hard to pierce 'sonic wall' precluding huge leaps in random 4K reading.



#40
NoelC

NoelC

    Software Engineer

  • Member
  • PipPipPipPipPipPipPip
  • 1,805 posts
  • Joined 08-April 13
  • OS:Windows 8.1 x64
  • Country: Country Flag

You're right, 4K is where it's at to improve responsiveness.

 

Re: the wall, I've always figured what must be happening is that the entire system takes a certain finite minimum amount of time to do an I/O.  At 4K bytes per chunk, you have to complete tens of thousand I/O operations per second to get better than 40 MB/sec throughgput.  That means whole operations have to complete, end-to-end, soup-to-nuts, getting 4K of data in and out literally in 100 microseconds or less (and do it over and over again consistently, not just once). 

 

As one who was around when microprocessors could execute only four or five machine instructions in that time, I always find that impressive in a relative sense.

 

The key of course to getting great small I/O performance is to minimize the latency added by the drive hardware itself, and that's what modern SSDs do extremely well. 

 

The Intel NVMe tech, on the other hand is apparently supposed to help with the latency in the pipeline.  I don't think we've seen all that it has to offer just yet, and I suspect it's going to help with that "sonic wall".  Perhaps Intel is purposely holding back, figuring as long as they can improve the speed and post bigger and bigger performance numbers, they won't need to drop the price.  Marketing.  No sense in showing their hand all at once.  Every new technology always benefits from optimization.  Just look at the differences between the drivers supplied by Windows and the latest versions downloaded from Intel.

 

By the way, a user on the OCZ forum not too long ago posted ATTO results that showed his RAID 0 array of Vector SSDs, using a modern motherboard and Intel RST subsystem (which caches low level I/O extremely effectively) delivering over 800 MB/second for 4K I/O.  ATTO uses Queue Depth 4 by default.  Edit:  I'm still trying to find that info.

 

-Noel


Edited by NoelC, 07 June 2014 - 11:34 AM.


#41
bphlpt

bphlpt

    MSFN Addict

  • Member
  • PipPipPipPipPipPipPip
  • 1,915 posts
  • Joined 12-May 07
  • OS:none specified
  • Country: Country Flag

NoelC, I know this is OT, but I was wondering ...
 
Since you admit that Win8.x is slower than Win7SP1 and there is not really much, if anything, that is a real improvement for actual work in Win8.x, and you have had to add several third party apps to Win8.x to bring back the look and feel of Win7SP1 to the desktop that seems generally preferred, have you considered doing a new install of an updated Win7SP1, complete with all of your old tweaks and anything new you have learned over the last two years along with updated versions of all the apps that you normally use day-to-day, maybe even set your system up to be able to dual boot Win7 and Win8.x?  You could then truly show apples to apples performance comparisons, say which apps perform better with one OS over the other, and maybe even make a few updates to the Win7 version of your book if appropriate, since I'm sure that Win7 will continue to have life and followers for quite awhile.  I'm sure that xpclient could also add comments about the comparison as well. Just a thought since you seem to be about the most capable person I know to do such a comparison on real metal.  I mean I know your are busy with your actual job and RL, but in your spare time ... heh-heh :)

 

NOTE: You might have already made such an installation, but I don't know if you have gone back and updated your Win7 book accordingly, if anything new was found, that is.
 
[ /OT ]
 
Cheers and Regards my friend


Edited by bphlpt, 07 June 2014 - 05:07 PM.

Posted Image


#42
NoelC

NoelC

    Software Engineer

  • Member
  • PipPipPipPipPipPipPip
  • 1,805 posts
  • Joined 08-April 13
  • OS:Windows 8.1 x64
  • Country: Country Flag

I'll come right out and say that there is literally NOTHING I've found essential in Win 8.1 that I couldn't have or do in Win 7.  Only thing is, it's current - and that may matter for things like security fixes, driver updates, etc.

 

I still maintain a VM that's the best setup I had done with Win 7, as configured per the best practices I listed in my book, and I have kept the system up to date.  I ran it just a little while ago to ensure that even with the latest updates the file system is a fair bit faster than Win 8.1.  I also maintain a Win 8.1 VM that's just like my host system, with an identical virtual hardware setup and configured to the best of my abilities, so I've been able to do some pretty careful head to head comparisons.  I don't think they're any less valid or useful than two different systems running on the same hardware, but with necessarily different hardware drivers.

 

In short, my well set up Win 7 VM is a good bit faster than my well set up Win 8.1 VM.

 

I haven't had to tweak my Win 7 book much in the past few years because it is essentially complete - it takes one from an out-of-box experience to a fully functional, well-tuned workhorse of a system.  Since I don't use Win 7 any longer for everyday work, I probably don't pay as much attention to the subtleties of Win 7, and I've put more effort into making my Win 8 book better, though to be fair I HAVE added a few sections to the older book in the not too distant past.  Looking at the market now - with Win 8 basically flopping badly - I probably SHOULD put some more effort into the older volume.  Thank you for the suggestions.

 

-Noel



#43
TELVM

TELVM

    Advanced Member

  • Member
  • PipPipPip
  • 349 posts
  • Joined 09-February 12
  • OS:Windows 7 x64
  • Country: Country Flag

... By the way, a user on the OCZ forum not too long ago posted ATTO results that showed his RAID 0 array of Vector SSDs, using a modern motherboard and Intel RST subsystem (which caches low level I/O extremely effectively) delivering over 800 MB/second for 4K I/O.  ATTO uses Queue Depth 4 by default.  Edit:  I'm still trying to find that info.

 

I'd expect that setup to hit the same 'sonic wall' @ 4K random read QD1.

 

4x Samsung 840 Pro (among the fastest consumer SATA III SSDs) in raid 0:

 

2a5f3152_as-ssd-benchIntelRaid0Vol6.2.20

 

^ Everything scales up nicely except 4K random read QD1, which stalls at almost exactly the same point that the Intel P3700 beast above.

 

 

 

"4K random QD1:

 

At the low queue depths you normally encounter in a desktop environment, all of these configurations perform fairly similarly. In fact, the striped setups are even a bit slower than single drives. This is because we're taxing the NAND flash's throughput. Parallelism is needed to distribute the workload across multiple dies on multiple channels.

 

4K random QD64:

 

Once we jump up into very high queue depths, both RAID-based arrangements distinguish themselves. It's only a shame that this is very atypical of any desktop workload, so you won't see it unless you take the 840 Pros into a more enterprise application.

 

... ...

 

... Striped drives are certainly better equipped to push more IOPS, but only when you're stacking commands more than four high. Jumping up to a queue depth of 32, 16, or even eight is really uncommon in a desktop or workstation environment. As a result, the performance differences are far less pronounced in the real world.

 

One SSD on its own scores again in the contrived tests we put together. The performance differences when we boot up and shut down Windows 8, then fire up different applications, are marginal at best and not noticeable in practice. Single drives actually manage to outperform the striped arrays some of the time, even ..."

 

 

Uncle Tom's - SSDs Raid 0: Great for becnhmarks, not so much in the real world



#44
NoelC

NoelC

    Software Engineer

  • Member
  • PipPipPipPipPipPipPip
  • 1,805 posts
  • Joined 08-April 13
  • OS:Windows 8.1 x64
  • Country: Country Flag

 

I'd expect that setup to hit the same 'sonic wall' @ 4K random read QD1.

 

That's the speed of modern computers for you.  Hey, I envy 35.98 MB/sec since I get about 26.

 

Note in the AS-SSD output above the small latency times reported - mere tens of microseconds.  And of course the write times are representative of no-wait RAM write-back cache, which tends to batch the writes together to the actual drive.  And speaking of cache, don't discount the effect it has on both writing and reading.  Benchmarks don't actually do that good a job showing you how well it works in real-world operations.  The bottom line is:  If you have a fast I/O system and lots of RAM, you will have a very responsive system for real world operations even if your 4K QD1 random I/O benchmarks aren't gargantuan.

 

Interesting read, that Tom's article.  Thanks for the link.

 

People sometimes don't realize that SSDs generally have so much higher MTBFs than HDDs that a RAID array of SSDs can be as reliable as a single electro-mechanical HDD.

 

And everyone should be doing backups.

 

After 2 years of practical experience I'm glad I went in the direction I did (4 SSD RAID 0) and would tend to do it again today (except I'd start with modern drives with low latencies).  Next system I put together I'll certainly consider those Intel boards, though.  It will be interesting to see what Samsung comes out with in that arena in the near future.

 

-Noel



#45
jaclaz

jaclaz

    The Finder

  • Developer
  • 15,176 posts
  • Joined 23-July 04
  • OS:none specified
  • Country: Country Flag

People sometimes don't realize that SSDs generally have so much higher MTBFs than HDDs that a RAID array of SSDs can be as reliable as a single electro-mechanical HDD.

Care to expand/detail on this? :unsure:
Like some actual MTBF comparison, detail on the Raid level used, etc?

And everyone should be doing backups.

Sure :yes:, the three golden Rules (JFYI):
http://www.msfn.org/...-hdds/?p=930329

jaclaz

#46
NoelC

NoelC

    Software Engineer

  • Member
  • PipPipPipPipPipPipPip
  • 1,805 posts
  • Joined 08-April 13
  • OS:Windows 8.1 x64
  • Country: Country Flag

Well, I don't have an analysis suitable for a Tom's Hardware article at hand, but the SSDs I bought have a 2 million hour MTBF.

 

Even enterprise-class HDDs at the time (e.g., Western Digital RE4, of which I have two) were only pushing 1 million hour MTBF, with consumer drives presumably less.  Let's say a half million hours in round numbers.

 

I roughly figured RAID 0 with any single drive failure causing loss of the whole array:  2 million hours MTBF for any one drive / 4 drives in the array = a half million hours MTBF for the array.  Roughly the same reliability as a typical HDD.

 

And it's just common sense.  A solid state device seems less likely to fail than an electromechanical one.  Beyond that, consider that HDDs use more power and get a good bit hotter than SSDs.  Heat kills electronics.  My fans almost never increase above idle.

 

Keeping in mind RAID 0 spreads the write load across several drives, one has only to make the array out of sufficiently large SSDs so that NAND wear is not a practical issue.   I figured mine would last 10 years at the write load I use before NAND wear becomes an issue - and I use my system pretty heavily.

 

-Noel



#47
jaclaz

jaclaz

    The Finder

  • Developer
  • 15,176 posts
  • Joined 23-July 04
  • OS:none specified
  • Country: Country Flag

I asked because I initially understood you previously quoted sentence backwards (not understanding at first sight that you meant specifically a RAID 0 setup ), all in all (partially) my bad  :blushing: .

If a HD MTBF=1,000,000, and a SSD MTBF=2,000,000, a 4 device RAID 0 of SSD's has approximately 1/4, i.e. 500,000, i.e. 1/2 of the single drive, now that you cleared how you half the MTBF of the hard disk before everything is clear.

 

As a side note there are however IMHO few things as inaccurate (or as wrongly perceived, you choose) as the rated MTBF, I have found flippism as reliable as SMART technology, and I would say that roulette playing should be almost as reliable as trusting the MTBF declared by the manufacturer :w00t: :ph34r:.

http://www.dailytech...article6404.htm

 

I personally consider MTBF one of the many almost meaningless metrics around (i.e. only useful - maybe - for comparisons but not in any way a method to appreciate a "real life" measure).  

 

 

jaclaz



#48
NoelC

NoelC

    Software Engineer

  • Member
  • PipPipPipPipPipPipPip
  • 1,805 posts
  • Joined 08-April 13
  • OS:Windows 8.1 x64
  • Country: Country Flag

Well, a correctly derived MTBF is meaningful - but maybe less so in the consumer world since Marketing seems to drive everything now and they're comfortable with outright lying.  And who's going to actually test a whole bunch of parts to see if a 2 million hour MTBF is accurate?

 

I personally believe consumer HDDs have MUCH less reliability (probably much less than half the true MTBF) as compared to enterprise class HDDs.  That's a lifetime of experience talking.  When I started buying enterprise class drives I stopped having data loss.  And there are some indications that in this day and age of "buying on price alone" things have gotten worse.

 

The brand of SSD drives I bought (OCZ) is by a company that makes enterprise class hardware as well, and the fruits of my research (which went a bit beyond reading the datasheets) implied their hardware is quite reliable (not to mention well-supported).

 

Bottom line is I've gotten 2 years and 2 months of reliable service out of 4 OCZ Vertex 3 drives under reasonably intense daily use.  I'd buy the brand again, on the assumption that their current hardware is as well-designed and manufactured. 

 

That being said, modern Samsung models sport slightly better numbers and since Samsung manufactures flash themselves, slightly better prices.  Specifications always tempt...

 

-Noel


Edited by NoelC, 08 June 2014 - 09:37 AM.


#49
jaclaz

jaclaz

    The Finder

  • Developer
  • 15,176 posts
  • Joined 23-July 04
  • OS:none specified
  • Country: Country Flag

Sure :), and I do believe in your results and do not even think of doubting about your choices, which surely were taken after some good research and thinking :yes:, I had however the impression (possibly false) that in that post you attempted to rationalize :w00t: your choices using as base the MTBF (which is as said IMHO in itself largely a meaningless metric).

 

I.e. it seems to me like you did most probably the right choice, but you did that because of your lifetime experience and (inborn or acquired) common sense and not because of the published MTBF's. or these played however a minor role in your choices.

 

jaclaz



#50
NoelC

NoelC

    Software Engineer

  • Member
  • PipPipPipPipPipPipPip
  • 1,805 posts
  • Joined 08-April 13
  • OS:Windows 8.1 x64
  • Country: Country Flag

Not exclusively on MTBF, no.  I try to use all the knowledge and wisdom I have accumulated to make choices.  And I try to share that knowledge when I can.  That's one of the reasons I love forums like this one.  It's also one of the reasons I started this thread - because I'm always looking for ways to improve that knowledge.  Regular, ongoing benchmarking is one way I keep in touch with how things are working.  Simply put, I watch for problems / trends using objective data.

 

Regarding the use of published MTBF...  It's one input, and like it or not bigger numbers are generally better.  They tend to indicate how confident the manufacturer is in their products, and are arguably more meaningful with higher-end products, which in an engineering company I tend to be involved with.

 

This round of questioning started with you asking me for specifics about my comment on an SSD array being as reliable as an HDD.  I figured quoting numbers as one way to justify the comment made sense.  But you're right, it's VERY difficult to compare apples to apples when considering different technologies.

 

-Noel






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users