[ipac] HIP and corrupt indexes

[Hui, Cecilia]

Casey,

You beat me big time. Not that size really matters, but you are on 6U vs
us 2U. But seriously, I am glad you guys shared the number with us.
Otherwise, I would think mine was pretty fast. Now I know I need
improvement.

Thanks,

cecilia

From: ipac-bounces at lists.tblc.org [mailto:ipac-bounces at lists.tblc.org]
On Behalf Of Casey Durfee
Sent: Monday, April 23, 2007 1:16 PM
To: ipac at lists.tblc.org
Subject: [ipac] HIP and corrupt indexes

 

I'm assuming you're running the indexer/HIP on Slowlaris?  The indexer
always was about 5-10x slower on Solaris than on Windows.  (No idea on
Linux speed, though).

 

Beyond your choice of OS for the indexer, the rate limiting step is the
speed of your database server.  We used to get about 30-50 records/sec.
against our decrepit old Horizon server and now get about 100-120
records/sec. against our shiny new one (Dell 6850/4x dual core Xeon/32
GB RAM/Sybase+Red Hat).  


>>> ipac-request at lists.tblc.org> 4/23/2007 12:44 PM >>
<mailto:ipac-request at lists.tblc.org%3e%204/23/2007%2012:44%20PM%20%3e%3e
> 


------------------------------

Message: 3
Date: Mon, 23 Apr 2007 15:28:40 -0400
From: Jonathan Rochkind <rochkind at jhu.edu>
Subject: Re: [ipac] HIP and corrupt indexes
To: Vaughn Stamper <VStamper at Ci.Hickory.NC.US>
Cc: "Dynix's Horizon Information Portal,formerly iPac \(discussion\)"
<ipac at lists.tblc.org>
Message-ID: <462D08E8.5000908 at jhu.edu>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Around 6 million bibs.  Our indexer does around 12 bibs per second, 
according to the output from the mass indexer where it updates you as to

it's throughput. (it's just bibs that HIP is indexing, not items, right?

Hmm, I guess that's not right. I'm confused. Still we'll estimate with 
bibs---although if it really were 6 million at 12/second, it should take

us even MORE than 3 days. So I can't explain exactly what's going on).

150,000 bibs in 40 minutes is about 60 bibs per second, which is a LOT 
faster throughput then I'm getting. Which confirms my own suspicion that

our HIP machine is really seriously underpowered--although the 
bottleneck could be our Horizon machine too, I guess. It's hard to say. 
I wish I knew more about how to measure/spec this stuff in order to make

a case that we need a faster HIP machine.

Question for anyone: When you run the mass indexer, what kind of 
throughput (records per second) is it reporting typically? And what kind

of machine do you have for HIP and/or Horizon, if you want to supply
that?

Jonathan

Vaughn Stamper wrote:
> Three days? Holy moley! I'm sure you've got a lot more items than we
do (~150,000), but ours completes in less than 40 minutes. The wait
until it reaches 50,000 items indexed so that the services can be
re-started is about 15 minutes.
>
>      - Vaughn

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.tblc.org/pipermail/ipac/attachments/20070423/99cc8dd4/attachment-0001.html