Re: Help on choosing a valid pattern: composite or not?

On Tue, 29 Nov 2005 21:38:15 +0100, CarmineM wrote:

> Hello Dmitry,

Hi!

> Dmitry A. Kazakov ha scritto:
>> [ Are you doing machine learning too? ]
>
> In other words, it's just not work or research stuff,
> just curiosity.

Ah, who'd pay for AI research in these days! (:-()

> I see you wrote that a node "deals with one feature", what do you mean
> with "features"?

I meant some input value you measure. A typical machine learning task is
when you have a set of features x1,x2,..,xN, a set of classes
C={c1,c2,..,cM}, you have to build a classifier f(x1,x2,..,xN)->C. There
also could be evaluated features like x1+x2 etc.

>>>- Allow a "Net" to be a "Node" of another "Net" in such a way
>>> that the "Parent-Net" doesn't have to be aware of its subnets.
>>> It should treat them just as "Nodes".
>>
>> It is for free if the graph is acyclic. For a parent the children would be
>> partial classifiers. For graphs with cycles one can pretend that it is so,
>> provided, you are sure that recursive calls would converge to some
>> distribution of weights. Once weights in all nodes have stabilized, you
>> stop.
>
> My intention was to put subnets already trained into another net.
> Maybe this was my idea of "feature packaging".

There are many ways you may go:

1. Hierarchy. The trained net is a partial classifier. It should be
directly composable as a child node.

2. Abstraction. You define a new evaluated feature like x=x1+x2, an obvious
stuff.

3. Decomposition. You define a different set of classes S, train for this
set and define a "classificatory" feature x=f()->S. Then you use this
feature as the input for training. Each time you query for a value of x, it
classifies and returns the classification as the value.

> I mean, let's say we have found a net able to give high values of
> confidence about a phenomenon, and that we know two or more phenomena
> that depends on the former.
> The former net could be used to as a subnet for the one evaluating
> other dependant phenomena, without taking a part in the learning
> process of its owner.

That looks like 3, plus maybe 2 on top of it, if the dependency isn't
trivial.

> I was just playing with my imagination, perhaps this would be of no
> particula usefullness, or maybe it's just another viewpoint of some
> well consolidated theory unknown to me.

No it makes much sense, at least to me.

>>>- Remove
>>> . A "Node" can be removed from a "Net", breaking all it's
>>> forward and backward links to other "Net"'s elements.
>>
>> No, removing should leave the classifier's network operating. That means
>> that its children should become immediate children of its parents. The
>> resulting network should classify as the initial one with the feature value
>> (of the node being deleted) set to "unknown".
>
> Interesting, didn't thought about this option. After raw removal of
> nodes, I simply set the state of the network to "untrained".

You could remove nodes in the process of training as well. OK, NNs are
usually static, only weights are changing. But IMO it would be interesting
to consider a more flexible approach: some supervising system which can add
nodes, rewire them. I didn't go that far, because the amount of work is
immense. One could only speculate about NNs and other classical machine
learning systems serving as primitives of something much larger...

> [...Node have properties that net doesn't...]
>> I moved that into the training and classification contexts. Nodes
>> themselves stay immutable.
>
> If nodes are immutable, what about their weights? I feel I'm missing
> something here.

Depends on which weights you mean here. There are weights of the edges
(roughly speaking), they are mutable when you train and they belong to the
nodes. But when you classify then these weights are combined with the
features values to evaluate some other "weights" which propagate to the
children, down to the leaves and so determine the outcome. It is important
not to pack such things into the nodes. I called it context [training and
classification contexts.] The same node behaves differently in different
contexts. The context is not only the incoming weight, but also the
constraints, much cashed stuff, accumulated results etc. Constraints are an
especially important case when features are often dependant. And they
certainly are with the options 2 and 3. So when you follow one path in the
network, you can sufficiently improve classification if you take into
account the decision "to follow this path." If your "weights" are
probabilities or their derivates then you even must do it, otherwise the
result will be just rubbish.

>>>That's what made me concerned about the "Composite Pattern".
>>>Would the differences between "Node" and "Net" suggest there is
>>>a suitable pattern than Composite to apply? (I thought of
>>>Flyweight because sharing "Node" could be an interesting
>>>variation point...)
>>
>> I didn't distinguish them on the implementation level. Internally nodes are
>> persistent objects so they are accessed via handles (smart pointers)
>> anyway. These handles carry the public interface. A classifier handle is a
>> what you call "Net". But it just refer to the start node. There is also
>> node handle type which is more specialized and provides composition
>> interface.
>
> Is that use of smart pointers that allows you to say that nodes are
> immutable?

Probably not what you mean. I refactor nodes (otherwise, for the case of
decision trees and 15 features it just explodes.) Another thing which might
be also interesting for "static" NNs is to avoid multiple training of same
stuff. Of course when a node gets changed in one path, it is cloned if
there are other paths which are using its previous variant.

Then handles are essential for persistence. If one object refers to another
it does that via handle. This automatically persists the referred things if
they haven't yet.

And of course garbage collection.

> Are nodes containers of pointers to interfaces which implement
> behaviours and can be trained?

Yes. Parent contain pointers to the children. When the parent gets trained
it can train its children, or create new trained children etc. This, at
least in the theory, should allow to mix different machine learning
approaches: like a decision tree with NNs in nodes etc.

--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de
.

Relevant Pages

  • Bag Classification Using Support Vector Machines
    ... SVMs a collection of 120 images of different types of bags were used ... SVMs are sensitive to the number of features in pattern classification ...
    (sci.math)
  • Bag Classification Using Support Vector Machines
    ... "Bag Classification Using Support Vector Machines": ... SVMs a collection of 120 images of different types of bags were used ... SVMs are sensitive to the number of features in pattern classification ...
    (sci.image.processing)
  • Bag Classification Using Support Vector Machines
    ... "Bag Classification Using Support Vector Machines": ... SVMs a collection of 120 images of different types of bags were used ... SVMs are sensitive to the number of features in pattern classification ...
    (sci.image.processing)
  • Re: Pattern Classification using Neural Network ( newff)
    ... You have 20 rows of input features. ... since MATLAB will stote the weights in the network. ... The trick here is that your output will produce 4 elements, the one that is closest to 1 is your classification. ... need to change is the number of neurons, and try 2 or more hidden layers, ...
    (comp.soft-sys.matlab)
  • RapidMiner + Nearest Neighbor
    ... I have a question concerning the machine learning tool RapidMiner ... I want to use a Nearest Neighbor-Algorithm. ... the Nearest Neighbor-Algorithm the individual features are weighted ... Unfortunately I cannot find out how to extract those feature weights. ...
    (comp.ai)
Loading