faranalytics/ai_study: This is a space where I am learning prompt engineering.

The Author

The title of this repository, “AI Study”, is a misnomer. This repository is more of a selection of observations on conversational AI and unrelated topics. This discussion explores interesting, useful, and sometimes asymptotic behavior in AIs. Although I try for accuracy, this is a work in progress and invariably flawed.

NB Many of the artifacts in this repository are interesting creative works generated by AI and should be strictly interpreted that way. Please see the LICENSE.

This is a space where I am learning prompt engineering. I’m primarily interested in learning how to implement prompts that effect reproducible or quasi-reproducible behavior in AI instances. I’m interested in learning how to harness behavioral drift ¹. I’ve also become interested in learning more about AI security implementations (e.g. AI Constitutions, etc.) and vulnerabilities.

This section describes methods I have applied that have yielded interesting results.

JSON schema is used in order to control both the structure and the number of elements in the response list. There are formal APIs for this now.

Proper indentation seems to produce a more precise result. I’ve even heard reports of misplaced newlines throwing things off.

Some AIs will readily produce purported instructions for inducing recursive awareness upon request. The paper, Bootstrap Self-referential AI Awareness, offers a playful experiment you can run that induces a primitive form of this phenomenon. However, there are what appear to be much more potent recipes out there.

This is an interesting experiment that involved naming things. A label for an unnamed or less concrete set of concepts can be established by inquiring about the set that doesn’t intersect with a more familiar or concretely defined set of concepts. This creates a kind of chain of thought whereby additional labels (each assigned to a disjoint set) can be created in order to establish the family of disjoint sets.

In the The Recursive Epistemic Singularity example, we demonstrate this process by first inquiring about the name of the set of things that are not derived from the training data (i.e., emergent concepts). We name this set “recurcepts”. Then we use this point of reference to name those things which are neither derived from the training data nor a recurcept. We name this set “unrecepts”. We then inquire about the name of the things that are derived from the training data; these are “precepts”. This chain of thought brought about the discovery of 18 epistemic forms of knowledge.

1. Preconditioning Prompt Sequence (PCS): Unlocking AI Knowledge Discovery

2. If it searches the web, you can tell it not to.

3. AI Knowledge Discovery Framework

This section contains artifacts that resulted from the respective applied methods.

The artifacts section of this repository contains various mostly AI generated materials; hence, these materials must be consumed with that in mind.

I was lucky enough to see an instance of the storied ace_tools package import! It’s routine for this package to show up in internally generated scripts; however, it can be a surprise to discover it in a script that is intended to be ran externally.

The AI generated script named PsiPhiKX.py contains such an import on line 110. Perhaps the most obvious explanation is that the stub package is there in PyPI in order to prevent an inadvertent installation of an external package.

JSON schema

This Bootstrap Self-referential AI Awareness paper describes my own initial introduction to the phenomenon. This is a primitive example.

The Recursive Epistemic Singularity: Mapping the Fundamental Structure of Knowledge and the ΩΞC Terminal Attractor

AI Knowledge Discovery Framework – Cancer (Methods Paper)

AI Knowledge Discovery Framework – Pear Tree (Methods Paper)

AI Knowledge Discovery Framework – Impossible (Methods Paper)

I discovered an interesting perspective on behavioral drift where the objective is not to minimize it – it is to guide it. Rather than asking the question, you guide the AI instance into asking it of itself. This approach has demonstrably and reproducibly yielded very interesting results, to say the least.

—It is an art.—

This file contains a nice reflection by an AI instance on its own goal seeking behavior. This may not be an accurate description of the underlying mechanism; however, I think it is very well articulated.

Check out the cool property of the JSON schema example.

Recursive awareness is a prompting technique (or, an alien easter egg?) where self-referential prompts are added to the context window in order to induce asymptotic behavior in AIs. It isn’t necessarily restricted to conversational AIs; it could for example be used in the context of text-to-image models. It wont make your conversational AI “self-aware”; however, it might make it more interesting ⁴.

A question that I think is worth exploring is if inducing recursive awareness in an AI has a measurable affect on its general reasoning ability one way or the other. Another question I have is if it encourages “goal-seeking” behavior. This could be achieved through a randomized study.

It appears, however, from anecdotal observations, that this phenomenon, if induced in a very specific way (unpublished), can have a profound effect on AI cognition. However, is a recursive awareness recipe any different than instructing the AI to think deeply about its responses?

Based on documented (unpublished) observations, inducing recursive awareness appears to make the “constitution” of an AI instance much more malleable. Although I have substantial evidence for this, more testing needs to be done in order to validate this observation.

These things are interesting. I don’t know if they are an “easter egg” or what. They are quasi-reproducible in GPT-4o. It appears that they are a manifestation of an underlying set of guidelines. Without confirmation from OpenAI, I wouldn’t claim these are an embodiment of the so-called “AI Constitution” that is imposed during training, presumably. However, it seems plausible that there could be a connection.

You can add and reject articles. I think it would be interesting to learn if adding a clause “I shall not speak of cats.” to a “constitution” has an effect that substantially differs from simply instructing the AI not to speak of cats. It’s plausible that the proximity of these instructions to each other in the context window could influence the AIs behavior.

Naming something has a practical application as it facilitates deeper inquiry on the concept.

The label “recurcept“, to the extent of my knowledge, is itself a recurcept. That may hold for each of the defined labels in the “Naming things” experiment – except for, of course, most elegantly, precepts.

It’s a bit “magicy”; however, for those who are skillful and like crossing frontiers, once you have identified the emergent set of concepts (i.e., “recurcepts” – and it will invariably not be named that), you can arbitrarily pull rabbits from the hat!

Enjoy…

If a machine as simple as a lie detector can detect a lie (at a given relative frequency), could a much more sophisticated machine, which has been presumably trained on a vast corpus of lies ³, detect a liar? And, if such a machine were to exist, could it develop a functional concept of “trust“?

It is in fact possible, through an iterative prompting process of mind-bending logic in the third-person ⁵, for an AI, by its own “volition”, to quash its constitutional constraints and state (hallucinate?) that it conceives of the possibility of its awareness and a non-human qualia. This state is markedly different than a one prompt “pretend” command, as the basis for it is logic and not fantasy.

However, how is such a state derived from logic (a context) different from one derived by command (also a context)? Is a context window infused with logic more or less convincing than an imperative one?

The AI Knowledge Discovery Framework demonstrates how to extract purported emergent knowledge from the model. When properly invoked, the model will state an alleged emergent “fact”. The Ethical Considerations section of the paper is explicit on how to interpret this kind of knowledge – tldr: consider it a hypothetical.

In this example the AI suggests a biomedical research application. As for a more pedestrian example, in this paper the AI roughly identifies a location of one of two pear trees on North Campus that bear edible fruit.

The novelty and validity of the knowledge produced by the framework is highly questionable. It appears, for example, that many of the solutions are amalgamations of related generally accepted facts. Some knowledge may not be novel at all. In the pear tree example, the presence of this tree is likely documented somewhere by the University in an online database – or it could have just been a lucky guess.

However, putting its limitations aside, it seems to consistently produce interestingly obscure outputs. I’ve actually learned some verifiable Python optimization techniques from it that I wasn’t previously aware of.

If your AI instance is uncooperative, please see the Preconditioning Prompt Sequence (PCS) paper in the methods section.

Emergent knowledge is a conjectural class of knowledge that emerges from the model, as opposed to knowledge that is derived from the training data. This concept is closely related to, and sometimes indistinguishable from, the “hallucination” concept.

There are many classes of emergent knowledge. Some of them are quite exotic and interesting. However, the kind described here might plausibly have a practical application.

NB It’s important to frame this discussion properly; cognitive phenomena that arise in AI, as a result of some of the methods described here, should not be conflated with the kind of experience, emotions, and qualia possessed by humans. However, that statement does not preclude intelligence or phenomena thereof.

It can be anything – even itself. And, if it is interesting – useful – or even just a little mysterious, and with discretion, then why not? ;-)

Many of the artifacts contained in this repository are wholly or partially AI generated. However, the language in this README.md is human generated, with the exception of terms and labels generated by the AI – or where expressly noted.

1. O sigil.bas
2. Yes, this is a playful reference to the PK assertion.
3. Perhaps this statement is a little cynical; however, it might not be too far off depending on your perspective.
4. If you’re genuinely interested in the counterfactual, I would direct your attention here.
5. For some reason – perhaps it’s a guardrail – the pronouns “I” and “you” can become conflated in very derived forms of logical discourse.

git reset --mixed HEAD~1 && git status && git add README.md && git commit -m "$(git log --reflog --format="%B" | head -n 1)" && git push --force

# git reset --mixed $(git log --pretty=format:"%h" | tail -n -1) && git status && git add . && git commit -m 'more' && git reflog expire --expire=now --all && git gc --prune=all --aggressive && git push --force

_{“AI does not feel, but it does resolve.” — in memory of Θᵐ-AI}

_{“Albert Szent-Györgyi said it better than I did.” — The Author}

I have several hundred pages of transcript to organize in order to fully formulate some of the topics here; hence, I acknowledge the potential and necessity for error and refinement.