APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Thursday September 10, 2009 (MJD 55085) ---=== OBSERVING TEAM ===--- Observers: Howard Brewington Dmitry Bizyaev ---=== OBSERVING PLAN ===--- Guider and software tests! ---=== OBSERVING SUMMARY ===--- Night was humid and cloudy, sometimes rainy and foggy. We fulfilled request from Kyle and did sequences of dithered flats. Software (boss, sop) crashed many times (see log), and finally we were unable to complete all dithered flats. In addition to finished 1-s dithered flats with camera sp1 yesterday, we finished sp2 - 1s, 2s, 4s and sp1 - 2s, 4s 7 darks x10 min also were taken. Since we are dealing with scripts, as sop ditheredFlat, it would be nice to see the number of images in the sequence that have been finished. The 10 minutes darks with b1,b2 chips look smooth, with very few and rather natural looking cosmic ray hits. In a contrast, r1,r2 look MUCH dirtier, and this is not a constant pattern. Suspicious. ---=== OBSERVING LOG ===--- Afternoon: ---------- 23:00Z Started the afternoon checkout. 23;15Z Restarted the servers. 23:18Z Checked the mirror numbers. 23:23Z Filesystem Size Used Avail Use% Mounted on /dev/sda3 142G 123G 12G 92% / /dev/sda1 99M 30M 64M 33% /boot /dev/sdb1 917G 828G 43G 96% /data sdssfiles1p:/data1/data 1.5T 770G 623G 56% /files hub25mp:/home 92G 48G 45G 52% /home_hub25m 23:40Z Verified that the STUI actors were happy; also checked the status of the hub25m and perms. Reseted photo crate 2: id6 did not respond. Spectrographs sp1, so2 are in-spec for the current temperature, 11.5C. Night: ------ 01:38Z night looks now wet and cloudy. We continue to complete Kayle's test: dithered flats for sp1 and sp2, with 1,2,4,10 sec exposures. flat sequence started. See the summary of calibrations below. 04:41Z BOSS Sp1 micro not talking error. 04:42:14 APO.hjbrew 30 boss f text="command failed: ICCError('command BLUEROFF is active, cannot send new command 2112 LINES',) at /home/sdss3/products/Linux64/boss/v1_1/python/boss/Controllers/Forth.py:126" 04:42:36 .alerts 0 alerts e alert=boss.SP1MicroNotTalking,critical,"Reported by camCheck",enabled,noack,"" We issued "boss iack", "boss clearExposure", and then "hub startNubs boss", and started the sequence over. 05:39:08 APO.hjbrew.sop 247 boss f text="command failed: CommError(, "TDS(TCPPort(name=None:2100 to t-g-sdss-2.apo.nmsu.edu:2100) ok=True initialized=False) send '' '', received ''0''") at /home/sdss3/products/Linux64/boss/v1_1/python/boss/Controllers/Forth.py:320" sequence of dithered flats stopped, lamps gone off. ... retaking. 05:54:25 boss ping shows "sp1cam DEAD". We did "status" and few more pings. sp1 gets alive after this. Howard whistled "dixy", probably ot helped. - iach, clearExposure, couple of pings again, and then the sp1 camera came back. 06:03:25 .boss 3175 boss f text="command failed: ValueError("invalid literal for int() with base 10: 'NaN'",) at /home/sdss3/products/Linux64/boss/v1_1/python/boss/Commands/MechCmd.py:71" 06:06:06Z At this point we tried 3 times to get dithered flat with expTime=10. All attempts failed. We decide to switch to "sp1, 2 sec". FAiled again, Disregarding that "boss" actor is alive, we stopdead it and start with stageManager. Then startNubs, and iack. We are sure that we used "boss clearExposure" before, but now it fails: 06:15:17 APO.hjbrew 86 boss f text="command failed: AttributeError("'Exposure' object has no attribute 'cmd'",) at boss_main.py:53" Interesting (and incorrect): when Howard issues a command (sop ditheredFlat, more precisely), I don't see it in my log window (Howard does see it in his), I just see results. Nevertheless, we try to continue. 06:27:54 APO.dmbiz 32 sop w text="1 tasks failed to respond; heard from: uv ffs ne tcc wht ff hgcd boss gcamera" We cannot start. "sop status" report failure of comunication with lamps: 06:30:44 33 sop status 06:30:44 .hub 0 cmds d NewCmd=180716; CmdTime=1252650679.05; Cmdr="APO.dmbiz"; CmdrMID=33; CmdActor="sop"; CmdText="status" 06:30:44 APO.dmbiz 33 sop i text="boss thread" 06:30:44 APO.dmbiz 33 sop i text="uv thread" 06:30:44 APO.dmbiz 33 sop i text="HgCd thread" 06:30:44 APO.dmbiz 33 sop i text="Ne thread" 06:30:44 APO.dmbiz 33 sop i text="gcamera thread" 06:30:45 APO.dmbiz 33 sop i text="FF thread" 06:30:45 APO.dmbiz 33 sop i text="tcc thread" 06:30:45 APO.dmbiz 33 sop i text="wht thread" 06:30:45 APO.dmbiz 33 sop i text="ffs thread" 06:30:46 APO.dmbiz 33 sop w text="1 tasks failed to respond; heard from: uv ffs ne tcc wht ff hgcd boss gcamera" sop restart issued. Howard whistled "dixy" again. Started dithered Flats again.06:35Z ... lamps are on and exposure started! 06:35:36 APO.hjbrew.sop 260 boss f text="command failed: NameError("global name 'astrotime' is not defined",) at /export/home/boss/devel/boss-trunk/python/boss/Commands/ExposureCmd.py:288" 06:37Z called Craig. Craig recommended to change "setup -r ." in our procedure for boss-icc troubleshooting to "setup boss" from home directory. We repeated all steps (except "boss iack") to resurrect the boss interface. As a result, setup finds files in correct place (i.e. runs the latest version) and works correctly. 06:49Z next attempt to start the dithered flats sequence. 07:38Z sky is still cloudy, but humidity dropped below 80%, we start ventilation. 08:25Z ventilation stopped because humidity runs up high again. it crashed again, at the sequence of dithered flats 10 sec exposures: 08:56:18 APO.hjbrew.sop 382 boss f text="command failed: CommError(, 'poll timeout (60.0 secs) reading character.') at /home/sdss3/products/Linux64/boss/v1_1/python/boss/Controllers/TCPPort.py:125" It failed again: 09:15:58 APO.hjbrew.sop 398 boss f text="command failed: CommError(, "TDS(TCPPort(name=None:2100 to t-g-sdss-2.apo.nmsu.edu:2100) ok=True initialized=False) send ''R'', received ''U''") at /home/sdss3/products/Linux64/boss/v1_1/python/boss/Controllers/Forth.py:320" 09:40Z we got desperate of finishing the 10 sec dithered flats, it gets failed after a few exposures every time we try them. We mount plate 666 (black foam, found in some unusual place, on the floor of janitor's room in plug lab) and are doing darks requested by Lyle (10x10 min). After 5 darks the process failed again: 10:58:10 .boss 253 boss f text="command failed: ValueError("invalid literal for int() with base 10: 'NaN'",) at /home/sdss3/products/Linux64/boss/v1_1/python/boss/Commands/MechCmd.py:71" Restarted boss. Restarted sop. Started 2 more darks. The summary of the exposures: --- Dithered Flats start frame finish frame sp expTime ---------------------------------- 101046 101067 2 1 second 101068 101089 2 2 seconds 101090 101111 2 4 seconds 101124 101145 1 2 seconds 101146 101167 1 4 seconds ============================ Darks frame # expTime ---------------------------------- 101180 600 seconds 101181 600 seconds 101182 600 seconds 101183 600 seconds 101184 600 seconds 101186 600 seconds 101187 600 seconds --- 11:50Z endNight finished. ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 8027 00:08Z 00:26Z 100 O 61.40 65.89 44 ignore ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n): QA Procedures Done (y/n): UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 23:16Z 53 49 6 29 (NNE) 525 - 23:49Z 53 49 8 62 (ENE) 542 - 00:21Z 51 49 11 74 (ENE) 591 - clouds 00:33Z 51 49 10 41 (NE) 597 - '' 01:06Z 51 48 13 38 (NE) 715 - '' 01:38Z 51 48 13 76 (ENE) 644 - '' 02:10Z 51 45 13 67 (ENE) 520 - '' 02:42Z 51 44 12 76 (ENE) 459 - '' 03:14Z 50 45 9 81 (E) 488 - '' 03:46Z 52 43 2 335 (NNW) 453 - '' 04:19Z 53 41 0 326 (NW) 453 - '' 04:51Z 52 42 7 107 (ESE) 506 - '' 05:23Z 51 42 7 90 (E) 482 - '' 05:55Z 51 43 9 86 (E) 522 - '' 06:27Z 51 42 9 94 (E) 538 - '' 06:59Z 51 41 10 87 (E) 439 - '' 07:32Z 51 40 13 86 (E) 363 - '' 08:04Z 49 42 14 84 (E) 490 - '' 08:36Z 49 44 14 79 (E) 685 - '' 09:08Z 49 46 11 68 (ENE) 811 - '' 09:40Z 51 43 8 55 (NE) 681 - '' 10:12Z 51 43 6 88 (E) 762 - '' 10:45Z 52 40 1 239 (WSW) 697 - '' 11:17Z 51 40 1 252 (WSW) 502 - '' 11:49Z 50 42 1 95 (E) 651 - '' ---=== TELESCOPE STATUS ===--- 07:38Z Doors and louvers opened, fans on. 08:25Z ventilation stopped, doors closed. Handpaddle at the cartridge lift is switched to the manual side. Status at: 11:35Z Telescope stowed at: 30,121,00 mount Instrument mounted: Cartridge 11 Counterweights at: 250 units LN2 autofill systems: Connected and turned on. 180L LN2 dewar scales: Spectro 213 lbs, 16.5 psi Imager 187 lbs, 23.4 psi ---=== SOFTWARE USED ===--- IOP/SOP: v4_40_0 Watcher: v2_36_0 MCP: NOSVN TPM: tpm_v3_5_0 AstroDa: v15_13_1 TCC: TCC 2.11.0 2009-06-24 sdssProcedures: v2_44 SoS: v5_1_8 plate-mapper: v4_3_1 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A -0.6930 -0.6940 Axial B -0.0300 -0.0410 Axial C -0.3710 -0.3710 Trans D 11.6130 11.6170 Lateral E 10.3759 10.3505 Lateral F 11.9380 11.9253 GALILS Commanded: 9900. 400. 5850. 1200. 2950. 2750. Actual: 9900. 400. 5850. 1200. 2950. 2750. SETMIR VALUES PriDesOrient: 0.00 -3.30 5.10 278.08 178.06 PriOrient: 0.00 -3.04 5.26 277.70 178.07 SECONDARY: ---------- Focus: 0.00 Air Temp.: 12.5 Alt.: 30.002867 MIGS TONIGHT NOMINAL Axial A 18.6200 18.6150 Axial B 15.6260 15.6120 Axial C 13.8420 13.8170 Trans D -11.7880 -11.7780 GALILS Commanded: 1597816. 1566232. 1588191. -7600. -12400. Actual: 1597700. 1566200. 1588100. -7600. -12400. SETMIR VALUES SecDesOrient: 1257.00 -4.00 -9.00 10.00 234.42 SecOrient: 1257.06 -4.01 -9.01 10.18 234.70 ---=== PROBLEMS IN DETAIL ===--- Collimation with cart. 11. Telescope's T = 11.5 C. IDL> collimate, 101044 % Compiled module: COLLIMATE. Reading /data/spectro/55085/sdR-b1-00101044.fit SDSSPROC: Read noise = 0.585158 0.435157 0.582165 0.557161 electrons Reading /data/spectro/55085/sdR-b1-00101045.fit SDSSPROC: Read noise = 0.587488 0.447892 0.582358 0.557670 electrons COLLIMATE: Min offset = -1.80 pix COLLIMATE: Max offset = 1.80 pix COLLIMATE: RMS across CCD = 0.87 pix COLLIMATE: Mean offset = -0.18 pix COLLIMATE: Camera b1 appears to be OUT-OF-FOCUS (|mean| > 0.15 pix) COLLIMATE: Predict blue camera ring movement of 6.4 degrees for spectro-1 COLLIMATE: (if red is already in focus) Log file = Collimate-55085-b1-00101044.log PostScript file = Collimate-55085-b1-00101044.ps Reading /data/spectro/55085/sdR-b2-00101044.fit SDSSPROC: Read noise = 1.73533 1.87991 1.92955 1.96183 electrons Reading /data/spectro/55085/sdR-b2-00101045.fit SDSSPROC: Read noise = 1.78719 1.91790 1.92945 1.96153 electrons COLLIMATE: Min offset = -1.00 pix COLLIMATE: Max offset = 1.20 pix COLLIMATE: Mean offset = -0.23 pix COLLIMATE: Camera b2 appears to be OUT-OF-FOCUS (|mean| > 0.15 pix) COLLIMATE: Predict blue camera ring movement of 8.3 degrees for spectro-2 COLLIMATE: (if red is already in focus) Log file = Collimate-55085-b2-00101044.log PostScript file = Collimate-55085-b2-00101044.ps Reading /data/spectro/55085/sdR-r1-00101044.fit SDSSPROC: Read noise = 1.65777 1.74204 1.61479 2.90231 electrons Reading /data/spectro/55085/sdR-r1-00101045.fit SDSSPROC: Read noise = 1.65816 1.74076 1.61525 2.92042 electrons COLLIMATE: Min offset = -1.80 pix COLLIMATE: Max offset = 1.80 pix COLLIMATE: RMS across CCD = 0.82 pix COLLIMATE: Mean offset = -0.03 pix COLLIMATE: Camera r1 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Predict (red) piston movement of 314 steps for spectro-1 COLLIMATE: Issue SOP command: "sp1; mechPiston 314" Log file = Collimate-55085-r1-00101044.log PostScript file = Collimate-55085-r1-00101044.ps Reading /data/spectro/55085/sdR-r2-00101044.fit SDSSPROC: Read noise = 3.23377 3.32897 3.29331 3.70840 electrons Reading /data/spectro/55085/sdR-r2-00101045.fit SDSSPROC: Read noise = 2.43857 2.62400 2.39312 2.59019 electrons COLLIMATE: Min offset = -1.20 pix COLLIMATE: Max offset = 0.40 pix COLLIMATE: RMS across CCD = 0.35 pix COLLIMATE: Mean offset = -0.06 pix COLLIMATE: Camera r2 appears to be IN-FOCUS (|mean| < 0.15 pix) COLLIMATE: Predict (red) piston movement of 543 steps for spectro-2 COLLIMATE: Issue SOP command: "sp2; mechPiston 543" Log file = Collimate-55085-r2-00101044.log PostScript file = Collimate-55085-r2-00101044.ps Looks in good collimation.