Written by Lucas Teske
on 19 February 2017

Saving the Raw Packet

Now that we have the handler for the demuxing, we will implement the function SavePacket. It will receive two arguments, the channel id and a packetdict. The channel id will be used for saving the packets in the correct folder (separating them from other channel packets). We may have also a Fill Packet here, that has an APID of 2047. We should drop the data if the apid is 2047. Usually the fill packets are only used to increase the likely hood of the header of packet starts on the start of channel data. So it “fills” the channel data to get the header in the next packet. It does not happen very often though.

In the last step we assembled a packet dict with this structure:

{
  "data": pdata,
  "version": version,
  "type": type,
  "apid": apid,
  "sequenceflag": SEQUENCE_FLAG_MAP[sequenceflag],
  "sequenceflag_int": sequenceflag,
  "packetnumber": packetnumber,
  "framesdropped": False,
  "size": packetlength
}

The data field have the data we need to save, the type says the type of packet (and also if its compressed), the sequenceflag says if the packet is:

0 => Continued Segment, if this packet belongs to a file that has been already started.
1 => First Segment, if this packet contains the start of the file
2 => Last Segment, if this packet contains the end of the file
3 => Single Data, if this packet contains the whole file

It also contains a packetnumber that we can use to check if we skip any packet (or lose).

The size parameter is the length of data field – 2 bytes. The two last bytes is the CRC of the packet. The CCSDS only specify the polynomial for the CRC, CRC-CCITT standard. I made a very small function based on a few C functions I found over the internet:

def CalcCRC(data):
  lsb = 0xFF
  msb = 0xFF
  for c in data:
      x = ord(c) ^ msb
      x ^= (x >> 4)
      msb = (lsb ^ (x >> 3) ^ (x << 4)) & 255
      lsb = (x ^ (x << 5)) & 255
  return (msb << 8) + lsb

def CheckCRC(data, crc):
  c = CalcCRC(data)
  if not c == crc:
    print "   Expected: %s Found %s" %(hex(crc), hex(c))
  return c == crc

On SavePacket function we should check the CRC to see if any data was corrupted or if we did any mistake. So we just check the CRC and then save the packet to a file (at least for now):

EXPORTCORRUPT = False
def SavePacket(channelid, packet):
  global totalCRCErrors
  global totalSavedPackets
  global tsize
  global isCompressed
  global pixels
  global startnum
  global endnum

  try:
    os.mkdir("channels/%s" %channelid)
  except:
    pass

  if packet["apid"] == 2047:
    print "  Fill Packet. Skipping"
    return

  datasize = len(packet["data"])

  if not datasize - 2 == packet["size"]: # CRC is the latest 2 bytes of the payload
    print "   WARNING: Packet Size does not match! Expected %s Found: %s" %(packet["size"], len(packet["data"]))
    if datasize - 2 > packet["size"]:
      datasize = packet["size"] + 2
      print "   WARNING: Trimming data to %s" % datasize

  data = packet["data"][:datasize-2]

  if packet["sequenceflag_int"] == 1:
    print "Starting packet %s_%s_%s.lrit"  % (packet["apid"], packet["version"], packet["packetnumber"])
    startnum = packet["packetnumber"]
  elif packet["sequenceflag_int"] == 2:
    print "Ending packet %s_%s_%s.lrit"  % (packet["apid"], packet["version"], packet["packetnumber"])
    endnum = packet["packetnumber"]
    if startnum == -1:
      print "Orphan Packet. Dropping"
      return
  elif packet["sequenceflag_int"] != 3 and startnum == -1:
    print "Orphan Packet. Dropping."
    return

  if packet["framesdropped"]:
    print "   WARNING: Some frames has been droped for this packet."
  filename = "channels/%s/%s_%s_%s.lrit" % (channelid, packet["apid"], packet["version"], packet["packetnumber"])
  print "- Saving packet to %s" %filename


  crc = packet["data"][datasize-2:datasize]
  if len(crc) == 2:
    crc = struct.unpack(">H", crc)[0]
    crc = CheckCRC(data, crc)
  else:
    crc = False
  if not crc:
    print "   WARNING: CRC does not match!"
    totalCRCErrors += 1

  if crc or (EXPORTCORRUPT and not crc):
    f = open(filename, "wb")
    f.write(data)
    f.close()

    totalSavedPackets += 1
  else:
    print "   Corrupted frame, skipping..."

With that you should be able to see a lot of files being out of your channel, each one being a packet. If you get the first packet (with the sequenceflag = 1), you will also have the Transport Layer header that contains the decompressed file size, and file number. We will handle the decompression and lrit file composition in next chapter. You can check the final code here:

https://github.com/racerxdl/open-satellite-project/blob/master/GOES/standalone/channeldecoder.py

← →

Top