Seeing red

[The Grand Nudger]

More musing over the mechanical problems between computational steps than the steps themselves, actually..particularly in the context of bennys question.   The issue of signal size, speed, and buffering explains for us both how a system can accommodate chaos -and- some experiences or behaviors we're all familiar with.  Like zoning out when we focus, or going to "lalaland".  

Obviously we must have a way to deal with this as it's an overwhelmingly present issue, mechanically, and your comments speak to that effect...I propose frontloading as a way of overcoming the issue..but frontloading doesn't work in the context of the nn's you're describing.  I propose frontloading architecture, specifically, not frontloading processes (which is what your NN's could do) because it means that it's a problem that exists, but doesn't matter to the architecture in question...it doesn't have to do anything in specific, or "learn" anything to overcome it.  

If a system -does- actually have to respond to the problem (rather than engineering it;s solution at the level of hardware), the it's an inevitability that it's going to run into a situation where it's trying a 40 inch peg down a 20 inch hole, and in context...I think we call that a stroke, lol.  We could say, well..it learns to build a process center for dealing with that, which is fine, as long as it learns that before it encounters the problem, that would work..until the inevitability of that process center encountering the same error -in itself-.  That's alot of room for failure in a system that is going to be subject to alot more "chaos" out in the "real world" than it will in sim environment where some things are taken for granted and others are completed ignored for purposes of brevity.

As for the programming, yep I see what you mean. The programming in this case probably comes about not just from general neural network dynamics but also from the specific connectivity patterns... different connectivity patterns will create different network dynamics... as for instance is the case with the bidirectional connectivity of the cortex... that creates completely different (and really cool   ) network dynamics than areas without that sort of connectivity.

-another interesting thing to note, is that whatever state the connected system is in is itself, by simple virtue of it's existence, is the default state of the machine language of the system, in the absence of other instructions it will always return to this state.  The more connected it is the more it can do -without- higher level translation and instructions, without programming.  For example, when you subject data to your alu it's actually performing all possible functions on that data by default..you're selecting which output you prefer. It's doing all that logical work without your input, you don't have to tell it to do that (that's how we sped up processors initially, made their default states encompass a wider range of work). That lessons the load on whatever processes you can imagine for it and even for a "learning system" it lightens the load of computation....it also gets around an interesting question as regards learning systems.  If they express their functions by the system of weights you have...how did they learn to learn?   Personally, I think that brains are a bastard amalgam between heuristics and dedicated functions..and I think they come that way standard, from birth.  That the architecture itself supports a minimum level of operation (and I'm talking high level cognition here, not just keeping the heart beating) by virtue of -nothing- other than it's construction.  No complex high level programming relationships or referential data sets saved from experience required.

What are your thoughts.  How does the NN learn to learn, and do you think that we're nn blanks at birth (or to what extent would we or could we be?), or do you expect that like a pc...we're shipped with bundled utilities, lol? Or do you see us more as hard built to solve for x.  A loaded gun waiting for the trigger to be pulled? Also, can you imagine a failure condition for an NN. Some scenario in which they are provided with the inputs but fail to comp? What would that be like, how could it happen?