EventStream.transformer.conditionally_independent_model module¶

The conditionally independent event stream GPT model.

class EventStream.transformer.conditionally_independent_model.CIPPTForGenerativeSequenceModeling(config: StructuredTransformerConfig)[source]¶

Bases: StructuredGenerationMixin, StructuredTransformerPreTrainedModel

The end-to-end model for conditionally independent generative sequence modelling.

This model is a subclass of StructuredTransformerPreTrainedModel and is designed for generative pre-training over “event-stream” data, with inputs in the form of PytorchBatch objects. It is trained to solve the generative, multivariate, masked temporal point process problem over the defined measurements in the input data.

This model largely simply passes the input data through a ConditionallyIndependentPointProcessTransformer followed by a ConditionallyIndependentGenerativeOutputLayer.

Parameters:¶

config: StructuredTransformerConfig¶

The overall model configuration.

Raises:¶

ValueError – If the model configuration does not indicate conditionally independent mode.

forward(batch: PytorchBatch, is_generation: bool = False, **kwargs) → GenerativeSequenceModelOutput[source]¶

This runs the full forward pass of the model.

Parameters:¶

batch: PytorchBatch¶

The batch of data to be transformed.

is_generation: bool = False¶

Whether or not the model is being used for generation.

**kwargs

Additional keyword arguments, which are used for output structuring and are forwarded to the encoder. The model specifically looks for use_cache, output_attentions, and output_hidden_states keyword arguments, which control whether additional properties should be added to the output.

Returns:¶

The output of the model, which is a GenerativeSequenceModelOutput object.

prepare_inputs_for_generation(batch: PytorchBatch, past: tuple | None = None, **kwargs) → dict[str, Any][source]¶

Returns model keyword arguments that have been modified for generation purposes.

Parameters:¶

batch: PytorchBatch¶

The batch of data to be transformed.

past: tuple | None = None¶

The past state of the model, if any. If specified, it must be a tuple containing the past values over prior layers and heads.

**kwargs

Additional keyword arguments. If “use_cache” is set in the kwargs to False, then the past state is ignored. If not, then the past state is passed through the model to accelerate generation, if past is not None then the batch is trimmed to the last element in the sequence, and the sequential attention mask is pre-computed.

Raises:¶

ValueError – If the past state is malformed or if there is a dep_graph_el_generation_target in the kwargs that is not None.

class EventStream.transformer.conditionally_independent_model.ConditionallyIndependentGenerativeOutputLayer(config: StructuredTransformerConfig)[source]¶

Bases: GenerativeOutputLayerBase

The output layer for the conditionally independent event stream model.

TODO(mmcdermott):

Allow for use of NLL-beta throughout? https://github.com/mmcdermott/EventStreamGPT/issues/26

Parameters:¶

config: StructuredTransformerConfig¶

The overall model configuration.

Raises:¶

ValueError – If the model configuration does not indicate conditionally independent mode.

forward(batch: PytorchBatch, encoded: FloatTensor, is_generation: bool = False) → GenerativeSequenceModelOutput[source]¶

Returns the overall model output for the input batch.

It takes the final hidden states from the encoder and runs them through various output layers to predict subsequent event timing and contents. It’s difference from a nested attention variant is largely in that it predicts everything simultaneously.

Parameters:¶

batch: PytorchBatch¶

The batch of data to process.

encoded: FloatTensor¶

The encoded representation of the input data.

is_generation: bool = False¶

Whether or not we are in generation mode. If so, the output predictions are for the next event for both time and event contents; if not, then we shift the event contents predictoin back by one event in order to align with the labels.