[ipac] HIP and corrupt indexes

[Vaughn Stamper]

Six million... wow. I knew it would be high, but wouldn't have guessed that.

For the record, we bought our HIP server in June, 2003. We did a clean install of Server 2003 on it when we migrated to Horizon in August, 2004, reinstalling HIP. Particulars: Dell PowerEdge 2600, dual 2.8GHz Xeons, 2 GB RAM, SCSI U320 15k RPM drives - RAID 5. Our database server came in August 2004 and is another 2600 also running S2K3 with dual 3.1 Xeons, 4 GB RAM and the SCSI U320 15k RPM drives running, RAID 1+0. We do get around 60 - 70 records per second.

Vaughn Stamper
Technology Coordinator
Hickory Public Library
Hickory, NC 28601-5126
828-304-0500 x239
http://www.HickoryGov.com/Library


-----Original Message-----
From: Jonathan Rochkind [mailto:rochkind at jhu.edu]
Sent: Monday, April 23, 2007 3:29 PM
To: Vaughn Stamper
Cc: Dynix's Horizon Information Portal, formerly iPac (discussion)
Subject: Re: [ipac] HIP and corrupt indexes


Around 6 million bibs.  Our indexer does around 12 bibs per second, 
according to the output from the mass indexer where it updates you as to 
it's throughput. (it's just bibs that HIP is indexing, not items, right? 
Hmm, I guess that's not right. I'm confused. Still we'll estimate with 
bibs---although if it really were 6 million at 12/second, it should take 
us even MORE than 3 days. So I can't explain exactly what's going on).

150,000 bibs in 40 minutes is about 60 bibs per second, which is a LOT 
faster throughput then I'm getting. Which confirms my own suspicion that 
our HIP machine is really seriously underpowered--although the 
bottleneck could be our Horizon machine too, I guess. It's hard to say. 
I wish I knew more about how to measure/spec this stuff in order to make 
a case that we need a faster HIP machine.

Question for anyone: When you run the mass indexer, what kind of 
throughput (records per second) is it reporting typically? And what kind 
of machine do you have for HIP and/or Horizon, if you want to supply that?

Jonathan

Vaughn Stamper wrote:
> Three days? Holy moley! I'm sure you've got a lot more items than we do (~150,000), but ours completes in less than 40 minutes. The wait until it reaches 50,000 items indexed so that the services can be re-started is about 15 minutes.
>
>      - Vaughn
>
>
> -----Original Message-----
> From: ipac-bounces at lists.tblc.org [mailto:ipac-bounces at lists.tblc.org]On
> Behalf Of Jonathan Rochkind
> Sent: Monday, April 23, 2007 12:05 PM
> To: Dynix's Horizon Information Portal, formerly iPac (discussion)
> Subject: Re: [ipac] HIP and corrupt indexes
>
>
> For those interested, the issue was a mis-configured value in either 
> word_index or matham on Horizon. Both tables have a value that should 
> point to the host name of the HIP configured.  But my 'test' Horizon 
> instance was pointing to a different HIP than I expected. Which made my 
> HIP running on the server I was looking at behave very strangely, in 
> ways that just didn't make any sense at all.  So in fact my HIP on 
> machine A was somehow using HIP indexes located on machine B. Or at 
> least that's what it appears to have been doing.
>
> I bet this means my reindex didn't work at all, and I have to start the 
> mass index again, and wait another three days. Sigh.
>
> Jonathan
>
> Jason Stephenson wrote:
>   
>> Jonathan Rochkind wrote:
>>
>>   
>>     
>>> And yet---they are. With the same bad results. This makes me think 
>>> there's some mysterious 'ghost' hzsearch process hanging around, somehow 
>>> still looking at the old indexes. I have no idea if this makes any sense 
>>> at all. Anyone have any advice?
>>>     
>>>       
>> Don't build the indexes in the background and see what happens. 
>> Basically, shut down all HIP services, then start the index server and 
>> mass indexer. After about 50,000 or so records have been indexed, 
>> restart the other HIP services.
>>
>> Reboot the machine between shut down and start of the index server if 
>> necessary.
>>
>> Cheers,
>> Jason
>> _______________________________________________
>> ipac mailing list
>> ipac at lists.tblc.org
>> http://lists.tblc.org/mailman/listinfo/ipac
>>
>>   
>>     
>
>   

-- 
Jonathan Rochkind
Sr. Programmer/Analyst
The Sheridan Libraries
Johns Hopkins University
410.516.8886 
rochkind (at) jhu.edu