CT Signals and Signal Detection Theory

[LadyNerdsky]

This went on the server too, but I want to throw it here to allow long-form answers:
I’ve got a CT theory question for discussion that’s been in the back of my mind and Candy’s meme on general bantz just reminded me.
A diagnostic system rarely has a 100% hit rate. Take a blood test for cancer as an example. Diagnostic results fall into 4 categories: a hit (blood test positive, person has cancer), a correct rejection (blood test negative, person doesn’t have cancer), a miss (blood test negative, but person DOES have cancer), and a false alarm (blood test positive, but person does NOT have cancer). It’s important for any diagnostic (and I’d place vultology as a “diagnostic”) to know not just the “hits” but the rate of false alarms too, as well as why they occurred in the first place.
Signal Detection Theory looks at the sensitivity, accuracy and bias of a test/instrument/diagnostic. If your hit rate is very high, your test/instrument/diagnostic is very sensitive and accurate. If there are many false alarms, too much “noise” was mistaken for “signal” and the test/instrument/diagnostic is highly biased. Many false alarms means the test is finding signals when there were none.
Why is this important to CT? The only people who hang around the forums, and the server, and contribute heavily to CT discussion are the “hits”. The people who pay for a typing and disagree with their results never to be heard from again are potentially the “false alarms”. I would argue for a test and theory to be complete and accurate, attention needs to also focus on the ones who feel their results based on vultology signals does not match their perceived behaviour/psychology.
This is a potential minefield, because why do these people feel their perceived behaviour/psychology doesn’t match anyway? Are they (1) lacking in accurate self-awareness, but their CT result is correct, or (2) self-aware and their CT result is incorrect? How would we ever know whether option (1) or (2) is correct? But, if their reading is indeed a “false alarm”, what information about the sensitivity and bias of the current set of CT signals can be gleaned? Is there a pattern to the mistypings? Is there a pattern to which signal or cluster of signals within the cognitive function are affecting accuracy? Should it/they still be a signal if it’s right 50% of the time? And so on.

[Juan E. Sandoval]

Spoiler

(wow, this became a mini article! @ladynerdsky lol - well either way, excited to hear your thoughts if you manage to read through my wall of text! ^^; i apologize that it became such a beast, and that i don't have time to widdle it down to anything more concise. hope it makes sense somehow, since i really like this topic and really want more member thoughts on it)

[collapse]

These are indeed big methodological hurdles in need of overcoming. And I'm glad you opened up this discussion-- as maybe it can help brainstorm better experimental control.
One problem we face is that, unlike SDT experiments, we don't know what constitutes the absolute accuracy of the typing -- so there is no clear "experimental control." With cancer or something of the sort, we can check to make sure the diagnostic tool got it right or wrong, but that's only based on a higher examination with higher accuracy that we're comparing the diagnostic tool to.
Indeed, do we compare CT's vultology against self-perception, and consider it a "miss" when there is a mismatch of self-perception to CT? That would elevate the self-perception above the signals, which has some strength and weaknesses.
Comparing Vultology against Self-Awareness
a) On one hand (and maybe liberalism and romanticism is largely to thank here), we are told we are the best judges of our own reality. That even though our knowledge is limited, nobody knows us better than ourselves, and that we probably have the clearest view of ourselves via introspection and personal autonomy-- if we so choose to exercise that self-awareness.
b) On the other hand, many experiments also confirm to us that we are poor judges of ourselves, perhaps even worse judges of ourselves than other people. We don't act out of our own conscious choice as much as we do out of an automatic brain activity which we only ever see the end sliver of -- as the thoughts populate into our minds. We don't know how we think, we just think. And some of us are self-reflective + keen enough to see the "how" but statistically speaking, most cannot do this right. And it's questionable whether this is due to a lack of exercised self-awareness or if humans simply fail, at a more fundamental level, to see their thoughts before they hit their minds.
~~~~~
I think the most scientific methodology to apply (in a kind of idealistic sense), would indeed be independent of self-perception, so that a person can be designated as [type X] using something like genetic testing or brain scans, despite what opinion of themselves they may have. We don't decide if we have cancer based on how we feel about ourselves-- if it's a fact, then it's a fact.
But so long as that so-called "fact" cannot be proven properly, this ideal scenario doesn't truly exist yet. Since we don't have genetic testing or brain scans, but instead an instrument that measures secondary effects of a hypothesized cognitive Type based on body mannerisms and facial expressions, the hit rate is indeed not 100%. And it may be that, even in the ideal scenario of a "perfected" vultological system, the hit rate still could not reach 100% the way genetic testing could. (i.e. paraplegics and comatose people who displaying no vultology still have types, but in such cases we can only know via non-vultological means)
In such a future scenario, vultology would then take a back seat to genetics/neuroscience, and be useful more in scenarios where genetic testing and neuroscience are not viable options. For example, if all we have is video footage of a now-deceased person.
~~~~~
Now, all of the above (while very fascinating to me) is not what I see most people think about or care about when they enter into typology or psychology in general. And this is very relevant because it does interfere with the research.
The Fuzzy Ground
The majority of people who pass by CT are interested in themselves through a personality angle, and are looking for a heuristic that can make sense of their struggles with life, their complications with work and family (etc) in new ways and offer helpful insights about what to do about it.
But we're mixing domains here.
If type really is an objective reality, then in what sense does it overlap with the above concerns and desires? Is CT (assuming it's a genetic reality) something that is designed to answer these very personal questions for people -- and does its lack of ability to do so represent a lack of reality to type? Naturally the answer is no, but when CT is judged based on its ability to give instrumental value in the form of personal insights and an ability to frame their life circumstances properly -- it is being measured by the wrong criteria. It would be like deciding if a person is/isn't a man or woman based on how good they are with machinery ("man") or how good they cook ("female") -- where a high score on either category would count as the typing.
In reality, being a male or female (as a biological fact) has a generally small bearing on a person's personality-- with males and females being more similar than different in everything-- so that even something as substantially obvious as this genetic confirmation does not translate over to certain "necessary" personality characteristics.
Would people who are looking to read a robust dissertation on their own psychology based on their maleness or womenness --  be comparable to those seeking a detailed dissertation of their psychology based on their type? (if type is a genetic reality, rather than a heuristic personality category) --maybe?
The hypothetical treatment of CT as something objective, places it in a somewhat different category of examination. It cannot be fairly examined by psychological comparison. The categorization of type, strictly speaking, is a vultological one -- so that a mistyping is based on a misapplication of the signal code -- not on a psychological mismatch with the subject.
As stated in the video submission product page Policy:
"Please also note that this vultology report may not match your self-typing according to other typological systems, and only represents a proper typing within the Cognitive Typology system. Also, while we aim to predict your personal psychology through your expressions, this report is primarily a service for the identification of your expressive profile. Our report may not necessarily match your self-perception and is chiefly meant to provide you with a window into how you gesture and how your expressive signature compares to that of other individuals in our ever-growing database."
But then Why Bother?
But then again, it also begs the question of -- why is this meaningful? Why would anyone pay to have their body mannerisms read, and compared to other people across the world who also have the same body mannerisms -- if it's not being accompanied by a psychological dissertation which can be either resonant or dissonant with the subject?
Well, even in such a scenario, I do think there is value in the CT service. Even when we take out the psychological component and make no commitment or claim of psychological parallels, there is a wealth of information in knowing what celebrities you move and gesture like. I can imagine some services compare one's face to that of other celebrities for a match -- like this one. In this sense, the service is still offering something of instrumental value to the buyer outside of just an affirmation of type category through CT's internal systemics.
But-- Misleading Marketing?
Still, although the report policy claims to not offer anything beyond a comparison of one's mannerisms to other people in the database -- other material on the website do purport to be able to highlight a correlation between body mannerisms and features of cognition.
The problem is that, at the highest level of academic rigor, this has not yet been verified. And while it does seem quite verifiable at a macro scale (i.e. if you look at all the Fe-leads as a whole, you see a big similarity when compared to the Te-leads as a whole) even at this early stage, the case of any individual person matching or mismatching a given profile description is indeed a bit of a hit or miss. The psychological features connected to vultology (in the CT model at present) are stronger and more robust at the macro level, while it's a bit less predictable at the individual level-- as is true of many emergent phenomenon.
And since all subjects are more intent on measuring the truth of their results at the individual level-- for reasons I fully sympathize with --the validity of the CT instrument can appear diminished in their eyes because of the mismatches they may feel at a personal scale, even though they may fit the data at a broader scale.
Obstacles to Alleviating This Issue / Possible Solutions
The complexity of the issue of perfect-profile-resonance (i'll shorten to PPR) is, in my opinion, quite astoundingly underappreciated as a computational problem.
A good example of this is in how we seem to notice when Apple's Siri doesn't interpret our words properly and it seems dumb, because we are treating it against a human/Turing-level standard. Or how self-driving cars are still not viable because the computational load for calculating all the variables on a street is enormous, even though in isolation they're incredibly sharp at identify most street objects.
The viability of a product-- at a personal scale-- coming from a computational technology, often has its reward father out in its cycle of maturation. And I imagine there are few computational tasks more intensive than that of describing human personality.
The end product would be an ideally tailored profile that captures a person's psychological quirks, their struggles, habits, lifestyles down to a level of a keen psychotherapist (or better)-- using as its measurement an analysis of body mannerisms. That is quite an astounding suggestion, and the complexity of this as-of-yet-unrealized feat would be truly awe inspiring, given the sophistication of human nature.
Yet the task at hand is not appreciated in this way. But lets examine firstly some reasons for why this is the current standard:
Barnum Effect
The typological community has been spoiled by decades of the Barnum Effect, which artificially increases the "accuracy" and "hit" rate of profiles by making generalized statements which feel very personal, but which are just part of human nature. They may experience higher PPR even though, in the objective sense, they are being given less info specific to them.
Ignoring the Obvious
Another factor is that we pay little mind to the common. We often don't care much about the obvious stuff. Going back to the computational problem, it's taken many years for A.I. to be able to identify a "cat" properly in a random photo (with random lighting/positions/angles/resolution/etc) -- and some people see this and go "so what?" while others realize and appreciate the amazing complexity in A.I.'s development to be able to identify even something like this.
Similarly, vultology (ctvc ~1.5) is able to identify the most baseline features properly (i.e. Ji-lead, Pe-lead) quite consistently through visual means, which represents quite a computational feat by itself and shaves months off psychotherapeutic work in minutes. But this can appear unimpressive to a subject who is looking for the last little bits of their "unknown", which they themselves have still been unable to unearth about themselves, despite years of introspection.
What they want, to feel that PPR, is insight that goes above and beyond what they've been able to identify of themselves. So even a system that can objectively get to ~80% of what they already know of themselves, would be found wanting, despite the feat which that may represent --in a similar way to how it's unimpressive for an A.I. program to identify cats objectively when we learned to do that at age 3.
The fact that vultology is a material/physical investigative tool, and can reach these accuracy/hit rates via non-psychological examinations -- while still imperfect-- is an enormous and underestimated achievement.
Now, I personally feel that little can be done about the current high standard for PPR that exists in the typology population. It is part of a larger commoditization that characterizes our present generation and future -- nor do I think this is really a problem.
Still, it's important to realize (just from the point of view of acknowledgement) that the problem of high level of adoption of results, is a different one than that of the system's proper functioning.
The "type results" are being measured against something that is not necessarily properly framed in terms of expectations. The task of giving people profile descriptions that resonate with them, in all their idiosyncratic life circumstances and life-pathways, is a much bigger problem than whether or not their type is rightly identified in se.
For example, cognitive type could conceivably be validated as true if the 4 function axes (Ni/Se, Ne/Si, Ti/Fe, Fi/Te) are found to have:

• a high cronbach's alpha level (inter-correlation-reliability) across the items listen inside the function pairs (i.e. giddy giggling w/ snarling smiles, etc)
• a high p-value correlation with behavioral features at a macro level
• a negative correlation (p-value) with the behavioral features of the hypothesized opposite pairing

Through these methods, via carefully controlled definitions of starting terms and what each signal means when treated as a motion-tracked vector, I think cognitive type could hypothetically --and optimistically-- achieve something like this and be validated scientifically, without producing a high PPR at the individual level since statistics rely more on aggregate results.
I make note of all of these things -- just to give a little bit of framework for how exactly CT, as an endeavor, truly collides with the world of pragmatic psychological diagnoses, the culture of personality typology, the culture of scientific inquiry, and so forth.
The Solution?
But in order for the CT results to meet the aforementioned high personality/typological expectations, the instrument would have to evolve above and beyond its technical requirements -- since the goal of science is not the same as the goal of psychotherapy. And CT is trying to be both science and psychotherapy.
The beginnings of this solution is already being established through the introduction of the psychoanalyrical system (i.e. dev levels). As I've said in Discord chat before, I don't think we can have PPR at a resolution lower than ~512 at a minimum.
What I mean by that is that, any system that tries to have a ~16 type summary of the human collective will have either:

• An incredibly high Barnum Effect
• An incredibly high miss rate

In the case of an incredibly high Barnum Effect, ~16 types can suffice in covering the whole population up until people really try to take it seriously. And in the case of a non-existent Barnum Effect (with very keen and measurable statements about types, rather than generalizations), the people that fall through the cracks is enormous.
There is no way to solve this problem except for an increase in resolution-- to begin approaching the legitimate diversity in humanity. Sixteen categories can never capture human nature beyond a very rudimentary level, and anyone seeking more keen and sophisticated insights about people will be disappointed by such a summary.
It's a computational problem, so the task would be to generate psychological profiles at that level of resolution-- while still being faithful to an empirical methodology-- and see whether or not that can properly increase the PPR.
My current hypothesis is that if we succeed in generating profiles at a resolution of 512 or 1024, while never leaving the empirical methodology, the PPR hit rate will improve dramatically. But until this is done, the newcomer subject has two main options for measuring the accuracy of their given type code:

• 1. Comparing it against the static function profiles presently available
• 2. Comparing it against the existing MBTI/Socionics corpus

In the first scenario, the hit rate may be good for the l--- types because the 8 function profiles are more 1:1 with that development, and as we saw together...

...the biggest chunk of types are l---. Still, collectively all non l--- developments add up to ~60% of people. So any non-standard development individuals reading the 8 function profiles will likely not resonate. For example, the TeSi l-ll Seelie may look at CT's own Te profile and say "that's totally not me." This is, at present, not a theoretical problem but a problem in the available resources. Theoretically, the existence of resources that could describe type down to the accuracy that we're currently measuring could do a lot more to increase the hit rate.
The 2nd option is of course the worst, where the newcomer subject compares their results to previous systems and ideas -- which are far off the mark. But if they happen to fall into that ~60% of people that are non l--- (which itself is a simplification since it doesn't account for differences in F attitude/gender even among l--- types), this second option is a recourse we can surely expect them to take.
So to properly measure the psychological hit rate of CT's real resolution, we need to build those modular 1024 profiles, otherwise people are comparing themselves against the wrong reference frames since function expression changes drastically with the combination of other developmental variables.

---

edit:
I realized this whole shpeel doesn't really answer your questions.. wrt Signal Detection Theory. I suppose I would summarize my point on that by saying that SDT would only properly work on vultology when vultology is being compared against genetics or neuroscience or some more well established metric of type and only after type's been verified. Otherwise we don't have a proper frame of reference for what is a miss/hit and who is ultimately right or wrong.
So I don't think SDT can be done between vultology and self-perception in any rigorous way at present, and the best we can do at the moment is have a dialectic methodology (vis-a-vis https://cognitivetype.com/cognitive-typology-methodology/ under "A Two-Way Relationship")-- where neither is treated as absolutely authoritative yet, and instead the effort becomes the investigation of the points of contact between the two fields-- and an ever strengthening focus on those points of similarity we find.

[Tea]

I've seen CT evolve to become more accurate than it was five years ago. Auburn weeds out much of the noise, though not with 100% accuracy. Some people's signals will be noisier than others because facial expression is multifactorial; it's not due to type alone. That said, there's still an essential clustering of signals, and from that, one can see past the noise. The centrality of a function never goes away.

[Author]

Posts