'Master.pl' is located at '/local/ice/'.
Usage:
Note: Master.pl uses /local/ice/saa_radar/ as a default directory for processing
files. The user can specify a different location (such as /cdrom to process
files from the cdrom drive) if they wish. If you wish to process data images
from the default directory, simply type 'Master.pl' without any command line
arguments.
Click HERE to see the script.
Click HERE to see the script.
Dependencies:
The SarAutoLib.pro script should also be located at the directory where
SarAuto.pro is run from. This is because it contains all of the functions
that the script calls to process each image. SarAutoLib.pro should be
compiled before running SarAuto.pro. This is because in order for
SarAuto.pro to call the functions to process each image, those functions
must first be compiled.
Click HERE to see an example of what this
script might look like.
ProcessHeaderInfo.pl lives at '/local/ice/'.
Click HERE to see the script.
UpdateWebPage.pl lives at '/local/ice/'.
Click HERE to see the script.
Click HERE to see the script.
Click HERE to see the script.
Process to make jpeg's and publish them to the RADARSAT web page:
SarAuto.pro should now be running and will process and publish all of the
images it can from the directory specified (or the default directory of
/local/ice/saa_radar/ if one wasn't specified) when Master.pl was run. No
further interaction is needed from the user.
Note: Any image sets that can not be processed will by moved to the holding
directory '/local/ice/RADARSAT_Holding/'. Right now, the reasons
why files might not be processed are if the file wasn't in the valid
RADARSAT format or all of the header information wasn't available in the
trailer file.
After that, it will check to make sure that the proper scripts are also
present. The scripts it checks for are ProcessHeaderInfo.pl and
SarAutoLib.pro. Without these scripts, the process will not be able to
complete since these scripts are called by the script, SarAuto.pro,
that Master.pl generates.
If everything is in order, the script then procedes to determine if
another location besides the default of /local/ice/saa_radar/ has been
supplied. This location is where the files are to be processed from.
Once the file location is set, a list of the zipped data files in that
directory is compiled.
For each data file in that list, Master.pl will determine if there are
matching leader and trailer files associated with that data file. If
those files are present, then we have a 'triad' (meaning we have a full
data set of the data, leader, and trailer files).
Once all of the files to process have been determined, Master.pl will
write the IDL script SarAuto.pro. This script will call the image
processing functions for each triad to be processed.
What happens when SarAutoLib.pro is compiled within IDL is that all of the
functions to process the image data will be compiled so that SarAuto.pro
can call them. If we didn't compile SarAutoLib.pro, SarAuto.pro would
return an error when trying to call the functions since they wouldn't
exist to IDL yet. This is because all of those functions are stored in
SarAutoLib.pro and weren't compiled.
When run, the first thing SarAuto.pro will do is unzip the data file to
process. It will then rename the data file to have a '.dat' extension and
the leader file to have a '.lea' extension. This is due to a problem
listed in the 'Current Issues' section of this document.
After that, it will call the ProcessHeaderInfo.pl script to parse out the
header information from the trailer file of the current triad being
processed. The data that is parsed out is the data that will be printed
on the picture when it is published to the web. This information includes
such things as the longitude and latitude the image shows, the time and
date it was taken, and other useful information.
From here, it will call the ProcessImageFile function located in
SarAutoLib.pro. This is the function that will do the actual processing
on the picture. The first thing the function does is to check to make
sure the image is a valid RADARSAT image. If not, the image will be moved
to a holding directory. (Currently /local/ice/RADARSAT_Holding). If it
can open the file, a directory under the web directory of
/local/ht-geo/great_lakes/icegroup/pics/(pic year)/(pic lake)/ will be
made. When that directory is made, the script will start processing
images and put them in the newly created directory. After all of the
different views, color tables, and subset pictures are made, the data and
leader files are renamed back to their original extensions and the triad
is moved off to a directory where they wait to be burned to CD for storage
purposes.
When all of the images have been processed, the UpdateWebPage.pl script is
called. This script will go through all of the year and lake directories
and will update the 'View Select' web page to reflect all of the processed
images. New links will be made so they will be available to the public
for viewing.
Congratulations - you've done it! Now it's time to sit back and relax and
bask in the glory of your work...
Resolution:
Resolution:
(todo: Add information about Houdini function to open the unopenable...)
Click HERE for the source code of this script.
Description:
Description:
Description:
Description:
Master.pl is a script written in PERL that dynamically builds the IDL
script, SarAuto.pro. It takes all of the image sets from the specified
directory (or the default directory if one is not specified) that have
existing data, leader, and trailer files and adds them to the SarAuto.pro
script to be processed. Nothing is actually processed until SarAuto.pro
is run. Master.pl simply builds the script.
Master.pl [Directory to process images from]
SarAutoLib.pro is an IDL script file that contains all of the functions
that do the actual processing on each image listed in SarAuto.pro. These
functions perform such tasks as making sure the data file is valid,
creating header information from the trailer file to add to the processed
jpeg image, constructing the actual jpeg, creating the different views and
filters of that jpeg image, and any other process that is applied to each
image. These functions are called for each image when SarAuto.pro is run.
SarAuto.pro is an IDL script that calls the functions located in
SarAutoLib.pro for each image to be processed. This script is generated
automatically when Master.pl is run.
The ProcessHeaderInfo.pl script should be located at the directory where
SarAuto.pro is run from. This is because SarAuto.pro calls the
ProcessHeaderInfo.pl script when it is run.
Note: This is a script generated by Master.pl and will change as
different files are processed...)
ProcessHeaderInfo.pl is a script written in PERL that is called from
'SarAuto.pro'. Its sole function is to parse out information about each
image from its trailer file that will be added to the processed jpeg.
Examples of this information include the date and time the image was
taken, the latitude and longitude it shows, pixel and line spacing, etc...
This script updates the
RADARSAT ScanSAR Archive
web pages with any
new pictures that have been processed by running the scripts to process
ScanSAR images. When SarAuto.pro is done processing images, it will
automatically call this script.
SarSubset.cgi is a cgi script (perl script run from the web page) that
will display a close-up view of a selected point on an image. It is
called when the user selects the 'Subset' option from a given lake / year
archive page. To view a subset, choose the 'View Select' and click 'Show
Image'. This will bring up a picture of the image. From here, simply
click on the portion of the image you would like to enlarge and let the
script do the rest.
SarPickView.cgi is a cgi script (perl script run from the web page) that will
display an image from a list of available filters / views of any image
that was processed. This script gets called when the user chooses an
option from the 'View Select' list box on any given lake / year archive
page.
Note: RunSar.pro is located at /local/ice/
(This 'flow chart' will describe how these scripts actually process images
and generate the web pages to hold them)
As described above, Master.pl is a perl script that generates the IDL
script, SarAuto.pro. When run, it first checks to make sure that it is
being run by the user 'ice'. If not, it will exit since 'ice' is the user
who has authority to the directories with the files to process and also
the directories where the HTML code will be written to. (As a side
note, all of the perl scripts that are run check to make sure they are
being run as user 'ice').
When RunSar.pro is run, it will compile SarAutoLib.pro and SarAuto.pro. After
those two are compiled, it will run SarAuto.pro. This is an automated
process. All the user has to do is type 'idl RunSar' for all of this to happen.
(Note: This is done by RunSar.pro but can also be done manually)
To do this manually, the steps are as follows:
1. Load idl by typing 'idl'
2. Compile the programs by typing '.compile SarAutoLib' and then '.compile
SarAuto'
3. Run SarAuto by typing 'SarAuto' at the idl prompt after the two files have
been compiled
To actually process files, the IDL script SarAuto.pro is run. In
order to run SarAuto.pro, the IDL script SarAutoLib.pro must first be
compiled.
When trying to open some data files, ENVI will pop up a dialog box
asking for the location of the leader file. This is because the file
extensions used by default in ENVI aren't quite set to find the leader
file automatically if it has a '.L' extension.
To solve this problem, rename the '.D' file to '.dat' and the '.L' file
to '.lea'. When opening the data file, envi will now be able to open
the leader file since they have '.dat' and '.lea' extensions instead of
'.D' and '.L'
Certain data files can not be opened by envi. This is because their
header sections are missing certain data such as the number of samples
and the number of lines the file has.
As it stands, the only working resolution to this problem is to run a
script against the data file and see if it is a valid file to be opened by
ENVI. This is accomplished by looking at the first 192 bytes in the file.
If the first byte is indexed as byte 0, you can tell if the file is valid
or not by looking at bytes 188, 189, 190, and 191. (Indexing the
beginning as byte 1, these four would be 189, 190, 191, and 192). These
four bytes hold the ascii values of the number of bytes (pixels) per
record of data. If these four bytes are 53, 54, 55, and 50 respectively,
then these ascii values translate to 5672. (In ascii, 53 -> 5, 54 -> 6, 55
-> 6, 50 -> 2). This means that there are 5672 bytes (pixels) per record
of data. If these four bytes are 0, then the file can not be opened. By
getting these four bytes and studying them, it can be determined whether
or not a datafile can be opened.
In order to delete an image set from the web page, simply delete the directory where the image
pictures are located under and then run the 'UpdateWebPage.pl' script again.
To do this:
The directories that hold the files to be burned to CD are located at
'/local/ice/tea/CDsToBurn/'.
In order to set up the scripts to add the next year, all you need to do it edit one
line in the 'UpdateWebPage.pl' script. The line you need to edit is near the top
and starts with '@yearArray = ....'. All you need to do is add the new year to the
start of the list and you're all set!
This is a list of files with unusally large file sizes...
Name
Size (bytes)
/local/ice/saa_radar/NSS.R1.GT.D00103.T100804.P23P117.P21P118.A.D.gz
130,410,378
/local/ice/saa_radar/NSS.R1.GT.D00104.T220223.P23P117.P21P118.A.D.gz
126,120,687
/local/ice/saa_radar/NSS.R1.GT.D00109.T110707.P38N077.P37N075.A.D.gz
239,413,142
/local/ice/saa_radar/NSS.R1.GT.D00109.T110729.P37N077.P35N075.A.D.gz
287,805,300
/local/ice/saa_radar/NSS.R1.GT.D98266.T031344.N24P033.N25P034.A.D.gz
139,891,138
/local/ice/saa_radar/NSS.R1.GT.D98266.T031358.N25P032.N26P034.A.D.gz
150,609,292
/local/ice/saa_radar/NSS.R1.GT.D98266.T031412.N25P032.N27P033.A.D.gz
133,753,807
File Size (bytes)
Number Lines
Number Samples
Data Size (Lines x Samples)
(File Size - Data Size)
24896252
4784
4999
23915216
981036
27845116
5520
4871
26887920
957196
28136252
5432
4999
27154568
981684
28176252
5440
4999
27194560
981692
28216252
5448
4999
27234552
981700
28296252
5464
4999
27314536
981716
28336252
5472
4999
27354528
981724
28376252
5480
4999
27394520
981732
28416252
5488
4999
27434512
981740
28456252
5496
4999
27474504
981748
28496252
5504
4999
27514496
981756
28536252
5512
4999
27554488
981764
28576252
5520
4999
27594480
981772
28616252
5528
4999
27634472
981780
28656252
5536
4999
27674464
981788
28696252
5544
4999
27714456
981796
28816252
5568
4999
27834432
981820
28856252
5576
4999
27874424
981828
29056252
5616
4999
28074384
981868
29096252
5624
4999
28114376
981876
29136252
5632
4999
28154368
981884
29616252
5728
4999
28634272
981980
32515392
3994
3972
15864168
16651224
37840252
6112
5999
36665888
1174364
37888252
6120
5999
36713880
1174372
42160252
6832
5999
40985168
1175084
42352252
6864
5999
41177136
1175116
43072252
6984
5999
41897016
1175236
50805316
5220
4776
24930720
25874596
50806392
5219
4777
24931163
25875229
50824424
5222
4776
24940272
25884152
58415868
5000
11247
56235000
2180868
63624462
7361
4262
31372582
32251880
87902652
7352
5899
43369448
44533204
95672956
7340
6431
47203540
48469416
150793432
8911
8369
74576159
76217273
155275664
8891
8637
76791567
78484097
157196652
8954
8683
77747582
79449070
160013544
9130
8670
79157100
80856444
Note - this is just a rough sketch of the observations
Samples = 4999
Lines differ by 8:
File size differs by 40K
40K / 8 = 5000. (4999 + 1)
Samples = 5999
Lines differ by 8:
File size differs by 48K
48K / 8 = 6000. (5999 + 1)