Transferring large raster datasets

When transferring raster data, consider how large the data is, if you can make a local copy or it is a remote copy, and what distribution media is available.

Raster data, such as raster datasets or raster catalogs, can be extremely large, with datasets reaching sizes of several terabytes; therefore, special considerations need to be made to move data that, as a unit, exceeds commonly used transport media.

If data is duplicated on-site, you can copy and paste the data. The benefits of copy and paste are that you do not need extra space to store the export files and it is fairly simple to do. If the raster dataset is stored in a file geodatabase, you can copy the entire file geodatabase folder to the transport media. However, if the file geodatabase contains other data that you do not want shared, you will need to create a new file geodatabase on the transport media and copy the raster dataset into the new file geodatabase.

If the raster dataset is stored in an ArcSDE geodatabase, you can create a file geodatabase on the transport media and copy the raster dataset to it. The benefit of using the file geodatabase to transfer data is it is ready to use. You can simply connect the transport media and use the data. You can also use the ArcSDE sdeexport command line tool to move your data. Because the file size of the raster object could be too large for your transport media, you may need to break up the data into volumes. You can use the following to create volumes:

sdeexport –X option creates volumes

Some DBMS vendors provide a detachable database file capability. Oracle uses transportable table spaces—for example, at Oracle 10g, the data files are inter-OS capable, and SQL Server uses detached files. Using detachable database file capabilities is much faster than loading an exported file.

The sdeexport –X option should be used if the export file size exceeds your transfer media's available storage space. The –X option specifies the maximum size of the export file volume. At ArcSDE 9.0, the maximum volume size is 4 GB. At 9.1, the volume size increased to the OS file limits. To determine the size of the export file that will be created, use the sderaster list operation with the –storage option.

The sdeexport command requires a large amount of sort space to export raster data; therefore, you will need to create temporary space that is approximately 4 percent of the size of the final export file. If sderaster –o list –storage returns a total of 100 GB for the export file, you will need to have 4 GB of temporary space available.

sdeexport example:

$ sdeexport –o create –t earth –f earth –X 4G
ArcSDE for <dbms> Mon Jul 19 16:06:36 PDT 2004
SDEX File Export         Administration Utility
---------------------------------------------------------
Exporting ArcSDE object to “earth" in SDEX 9.0 export format ...
Exporting table “earth".
  Spatial column "FOOTPRINT"
  Raster column "RASTER"
    1 features converted.
    1 features exported.

$ ls –l
total 5
-rw-rw-r--   1 sde    nuucp  4294967296 Aug  5 17:01 earth.000
-rw-rw-r--   1 sde    nuucp  4294967296 Aug  5 17:01 earth.001
-rw-rw-r--   1 sde    nuucp        9281019 Aug  5 17:01 earth.002

sdeimport example:

$ sdeimport –o create –t earth_at_15m –f earth –k earth_at_15m

ArcSDE 9.0 for <dbms>  Mon Jul 19 16:06:36 PDT 2004
SDEX File Import         Administration Utility
----------------------------------------------------------
Importing SDEX from earth.000 ...
  Importing spatial column "FOOTPRINT"
  Importing raster column "RASTER"
    1 record read.
    1 record stored.

You can use the DBMS export format as an alternative to moving larger raster objects from one location to another. Unless you move the entire ArcSDE instance, you will need to manually update the ArcSDE metadata tables when you import the data. The easiest way is to create an empty raster object, drop the tables of that object, and replace them with the imported ones.

SQL Server allows you to detach the database so the files of its file group may be copied to another location.

Using Oracle transportable tablespaces, you can export the metadata of a list of tablespaces to an export file. To do this, the data file associated with the tablespaces and the export files is copied to the new location and the export file is imported. The tables must be in read-only mode when they are exported and copied. This also requires the use of parameter files.