Another MEF FITS issue with copying data

[mgregg128]

When I copy a science (data) extension from a WFC3 MEF fits file to a single extension fits file, and then copy it back to the same extension in the same file, it appears that the header has changed in some way that causes subsequent tasks to fail.

Specifically, I am copying a science extension out to its own file, running LACosmic, then putting the cleaned data back in place in the original image file. Even though the data look good, and things like DS9 have no problem with the file, other tasks in drizzlepac gag on the file, apparently because the WCS has been altered in some way.

Is there a workaround? Or what am I doing wrong?

Thank you,
Michael

[olebole]

Can you provide a small but complete example that shows the problem?

[mgregg128]

Here is an example, with image.fits being the standard wfc3 MEF file, which is a single exposure and has lots of CRs. The image pixels are in extensions 1 and 4 (or SCI,1 and SCI,2).

imcopy image.fits[1] sci01.fits
imcopy image.fits[4] sci02.fits

Then I run LA_Cosmic on each sci0*.fits to remove the CRs. (Unfortunately I have to use the IDL version since LA_Cosmic in iraf needs tools from STSDAS. This may in fact be part of the problem, see below.)

Now copy these back into the original image.fits:

imcopy sci01.fits image.fits[1,overwrite]
imcopy sci02.fits image.fits[4,overwrite]

The actual pixel data appears to be the same size and format at each step.
The problem, I have come to realize as a result of this thread, is that LA_Cosmic is stripping all the header information of the single extension data file and replacing it with just a short skeleton header of just a few lines. When the CR-cleaned data get copied back to the original image, the headers also get copied.
So of course any task that needs the original detailed header information is going to fail.

Correcting this would be an IDL LA_Cosmic issue, of course.

Meanwhile, though, is there a way to use imcopy (or something else in iraf) to move just the pixel data from the CR-cleaned output back into the WFC3 MEF file, without also copying the headers?
Or, alternatively, a way to copy the header information from the original MEF file into the CR-cleaned file?

Thanks for the help.

[olebole]

Why are you using IRAF here at all? IMO Astropy should easily do the job. Like:

from astropy.io import fits

hdul = fits.open("image.fits")
hdul[1].write("sci01.fits") # extract the first extension to a separate file
# ... do lacosmic...
sci = fits.open("sci01.fits") # or whatever the result file is
hdul[1].data = sci[0].data # Replace the data, assuming the same shape
hdul[1].header.update(sci[0].header) # Update/add header values if needed
hdul.write("image.fits", overwrite=True) # Replace the original file, or write a new one
sci.close()
hdul.close()

I would really take any chance to move away from IRAF and to use modern tools instead of a lengthy search how this could be done there.

[mgregg128]

Excellent question. I am in fact in the process of transitioning (finally) much of my reductions away from iraf,
and now this operation too. It’s just that in the past, I have often done this CR-cleaning to help the tweakreg process, and
have many scripts to process lots of files at once, etc. This operation used to work fine in iraf, and the image manipulation and LACosmic
could all be in the same script.
You are correct, though, it is apparently and necessarily time to move on.

Thank you for the help.

[olebole]

What I don't understand here: lacosmic is in fact a Python package, and it should integrate very easily with your data:

from astropy.io import fits
from lacosmic import lacosmic

hdul = fits.open("image.fits")
hdul[1].data, _ = lacosmic(hdul[1].data, …)
hdul[4].data, _ = lacosmic(hdul[4].data, …)
hdul.write("image.fits", overwrite="True")

So you don't need to do temporary files or so; just open the FITS file, replace the data with the CR-removed data and write back. Invoking IRAF here seems very artificial.

[mgregg128]

You are totally correct! I am just now seriously transitioning to python, etc., and am simply (foolishly?) trying to use what I know and am familiar with.
There is also, I just found, a python notebook implementation of lacosmic that is a big improvement over what I used to do because it allows flagging of the CRs in the DQ extension rather than actual in-place correction. This will allow tweakreg to ignore them, and give a much cleaner result, and also allow astrodrizzle the same ability, making the whole drizzle exercise better. This eliminates the need to copy extensions out and back in (and further issues which I won't bother to go into at this point!) I haven't tried it yet, but it should be straightforward.
Again, I appreciate the help and the cajoling to move on...
Thank you.